2023-12-06T07:36:32.604 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T07:36:32.605 DEBUG:teuthology.run:Teuthology command: teuthology --name sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi --archive /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417 --owner scheduled_sseshasa@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} --verbose -- /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/orig.config.yaml 2023-12-06T07:36:32.644 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T07:36:32.732 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417 branch: wip-sseshasa-testing-2023-12-04-1848 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} email: null first_in_suite: false job_id: '7479417' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sseshasa-testing-2023-12-04-1848 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - \(SMALLER_PGP_NUM\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW OPS\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sseshasa-testing-2023-12-04-1848 sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 owner: scheduled_sseshasa@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5341 sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 sleep_before_teardown: 0 subset: 222/150000 suite: rados suite_branch: wip-sseshasa-testing-2023-12-04-1848 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQVssVISaQqOamGjBZt/mY8Q2tIklX0RUy5g1WQZzGogN+AXO2/DfEhuPAJ7R8ULDgF4tL4li1qOe2kmKwf8BQ= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvIqCUOjcCj8G+G4cMbYPHfTHYBX87fiDKJMbdetC3yteJVYM9dkUHa/psN7FRl95bRJ+DNpXqfLM5Zv/zGt+M= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - mon_recovery: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-06_06:16:39 tube: smithi user: sseshasa verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-06T07:36:32.732 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa; will attempt to use it 2023-12-06T07:36:32.733 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks 2023-12-06T07:36:32.733 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-06T07:36:32.734 INFO:teuthology.task.internal:Checking packages... 2023-12-06T07:36:32.758 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96fa2578c622ca9c169af6f426ca8b1616130e58' 2023-12-06T07:36:32.758 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-06T07:36:32.758 INFO:teuthology.packaging:ref: None 2023-12-06T07:36:32.758 INFO:teuthology.packaging:tag: None 2023-12-06T07:36:32.758 INFO:teuthology.packaging:branch: wip-sseshasa-testing-2023-12-04-1848 2023-12-06T07:36:32.758 INFO:teuthology.packaging:sha1: 96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T07:36:32.759 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-sseshasa-testing-2023-12-04-1848 2023-12-06T07:36:32.899 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7644.g96fa2578 2023-12-06T07:36:32.899 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-06T07:36:32.905 INFO:teuthology.task.internal:no buildpackages task found 2023-12-06T07:36:32.906 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-06T07:36:32.930 INFO:teuthology.task.internal:Saving configuration 2023-12-06T07:36:32.944 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-06T07:36:32.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T07:36:32.977 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 07:31:26.823882', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQVssVISaQqOamGjBZt/mY8Q2tIklX0RUy5g1WQZzGogN+AXO2/DfEhuPAJ7R8ULDgF4tL4li1qOe2kmKwf8BQ='} 2023-12-06T07:36:33.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 07:31:26.825339', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvIqCUOjcCj8G+G4cMbYPHfTHYBX87fiDKJMbdetC3yteJVYM9dkUHa/psN7FRl95bRJ+DNpXqfLM5Zv/zGt+M='} 2023-12-06T07:36:33.002 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-06T07:36:33.010 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-06T07:36:33.010 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-06T07:36:33.010 INFO:teuthology.run_tasks:Running task console_log... 2023-12-06T07:36:33.109 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2f02319ee0>, signals=[15]) 2023-12-06T07:36:33.109 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-06T07:36:33.116 INFO:teuthology.task.internal:Opening connections... 2023-12-06T07:36:33.117 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:36:33.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:36:33.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:36:33.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:36:33.283 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-06T07:36:33.290 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2023-12-06T07:36:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2023-12-06T07:36:33.308 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2023-12-06T07:36:33.365 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2023-12-06T07:36:33.365 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2023-12-06T07:36:33.365 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2023-12-06T07:36:33.365 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-06T07:36:33.366 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T07:36:33.367 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2023-12-06T07:36:33.390 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-12-06T07:36:33.407 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-12-06T07:36:33.407 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2023-12-06T07:36:33.465 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-06T07:36:33.466 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2023-12-06T07:36:33.466 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T07:36:33.466 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-06T07:36:33.466 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T07:36:33.466 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-12-06T07:36:33.489 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-06T07:36:33.498 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-06T07:36:33.513 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-06T07:36:33.513 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2023-12-06T07:36:33.516 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-12-06T07:36:33.532 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-06T07:36:33.538 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-06T07:36:33.538 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2023-12-06T07:36:33.573 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-12-06T07:36:33.872 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-06T07:36:33.927 INFO:teuthology.run_tasks:Running task kernel... 2023-12-06T07:36:33.943 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-06T07:36:33.943 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-06T07:36:33.943 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T07:36:33.944 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T07:36:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T07:36:33.961 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2023-12-06T07:36:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T07:36:33.963 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-06T07:36:34.018 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-361.el9.x86_64 2023-12-06T07:36:34.018 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-361.el9.x86_64 2023-12-06T07:36:34.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2023-12-06T07:36:34.020 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-361.el9.x86_64 2023-12-06T07:36:34.020 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-361.el9.x86_64 2023-12-06T07:36:34.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2023-12-06T07:36:36.513 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 7.2 MB/s | 6.4 MB 00:00 2023-12-06T07:36:36.610 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.4 MB 00:00 2023-12-06T07:36:38.518 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 2.3 MB/s | 2.0 MB 00:00 2023-12-06T07:36:38.914 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 1.5 MB/s | 2.0 MB 00:01 2023-12-06T07:36:40.662 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 7.9 MB 00:01 2023-12-06T07:36:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 7.9 MB 00:02 2023-12-06T07:36:45.437 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 18 MB 00:03 2023-12-06T07:36:45.733 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 18 MB 00:03 2023-12-06T07:36:49.520 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 5.8 MB 00:00 2023-12-06T07:36:50.938 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2023-12-06T07:36:51.108 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.8 MB 00:02 2023-12-06T07:36:52.542 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2023-12-06T07:36:52.706 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-12-06T07:36:56.168 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 20 MB 00:03 2023-12-06T07:36:57.430 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-12-06T07:36:59.695 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-06T07:36:59.695 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-06T07:36:59.797 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:36:59.809 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2023-12-06T07:36:59.810 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:36:59.811 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 96 M 2023-12-06T07:36:59.811 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 121 M 2023-12-06T07:36:59.811 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:37:00.909 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-12-06T07:37:02.148 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 2.7 MB/s | 5.6 MB 00:02 2023-12-06T07:37:03.165 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-06T07:37:03.165 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-06T07:37:03.310 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:37:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:37:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2023-12-06T07:37:03.318 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:37:03.318 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 96 M 2023-12-06T07:37:03.318 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 121 M 2023-12-06T07:37:03.318 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:37:05.385 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 2.9 MB/s | 5.6 MB 00:01 2023-12-06T07:37:08.005 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 2.6 MB/s | 20 MB 00:07 2023-12-06T07:37:13.634 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 2.0 MB/s | 20 MB 00:10 2023-12-06T07:37:20.211 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 1.9 MB/s | 38 MB 00:20 2023-12-06T07:37:20.503 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 1.8 MB/s | 32 MB 00:18 2023-12-06T07:37:20.504 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:37:20.504 INFO:teuthology.orchestra.run.smithi084.stdout:Total 4.6 MB/s | 96 MB 00:20 2023-12-06T07:37:21.366 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:37:21.732 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:37:21.732 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:37:22.907 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-core-5.14.0-390.el9.x86_6 1.8 MB/s | 32 MB 00:17 2023-12-06T07:37:23.012 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:37:23.013 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:37:23.249 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 1.9 MB/s | 38 MB 00:19 2023-12-06T07:37:23.249 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:37:23.250 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.8 MB/s | 96 MB 00:19 2023-12-06T07:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:37:24.481 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:37:24.481 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:37:25.907 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:37:26.549 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-06T07:37:26.670 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:37:28.691 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:37:29.325 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-06T07:37:29.401 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:37:29.474 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:37:29.574 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:37:32.208 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:37:34.189 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:37:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:37:37.028 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:37:39.027 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:07.570 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:08.050 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:08.050 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-06T07:38:08.050 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:38:08.051 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:38:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:38:08.469 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:38:08.999 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T07:38:09.144 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:09.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T07:38:09.145 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-06T07:38:09.145 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T07:38:09.145 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T07:38:09.145 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-06T07:38:09.145 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T07:38:09.146 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T07:38:09.263 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:09.263 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T07:38:09.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 68.el9 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2023-12-06T07:38:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8264254 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T07:38:09.298 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T07:38:09.299 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T07:38:09.299 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2023-12-06T07:38:09.299 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:38:09.299 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T07:38:09.300 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T07:38:09.300 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-390.el9.x86_64 2023-12-06T07:38:09.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T07:38:09.948 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2023-12-06T07:38:10.392 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:10.436 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-06T07:38:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-06T07:38:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:38:11.247 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:38:11.347 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-06T07:38:11.390 INFO:teuthology.orchestra.run.smithi084.stderr:done 2023-12-06T07:38:11.393 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2023-12-06T07:38:11.408 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.5xzahipBCm 2023-12-06T07:38:11.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5xzahipBCm 2023-12-06T07:38:11.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.5xzahipBCm 2023-12-06T07:38:11.610 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.5xzahipBCm is 6KB 2023-12-06T07:38:11.659 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.5xzahipBCm 2023-12-06T07:38:11.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T07:38:11.738 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-06T07:38:11.738 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-06T07:38:11.738 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-06T07:38:11.738 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-06T07:38:11.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.771 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T07:38:11.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2023-12-06T07:38:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:11.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T07:38:11.919 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-06T07:38:11.920 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T07:38:11.920 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T07:38:11.920 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-06T07:38:11.920 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T07:38:11.920 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T07:38:12.037 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-06T07:38:12.038 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-06T07:38:12.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2023-12-06T07:38:12.081 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2023-12-06T07:38:12.081 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-06T07:38:12.082 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:38:12.083 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T07:38:12.084 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T07:38:12.084 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-390.el9.x86_64 2023-12-06T07:38:12.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T07:38:12.695 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-12-06T07:38:14.094 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-06T07:38:14.131 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-12-06T07:38:14.133 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-12-06T07:38:14.149 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.CjIPufjlUd 2023-12-06T07:38:14.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CjIPufjlUd 2023-12-06T07:38:14.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.CjIPufjlUd 2023-12-06T07:38:14.352 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.CjIPufjlUd is 6KB 2023-12-06T07:38:14.401 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.CjIPufjlUd 2023-12-06T07:38:14.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T07:38:14.479 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-06T07:38:14.479 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-06T07:38:14.479 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-06T07:38:14.479 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-06T07:38:14.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-06T07:38:14.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-12-06T07:38:41.906 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-06T07:38:41.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:38:41.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:38:44.634 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-06T07:38:44.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:38:44.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:00.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-12-06T07:39:03.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-06T07:39:09.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:39:09.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:12.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:39:12.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:12.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-12-06T07:39:15.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-06T07:39:24.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:39:24.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:27.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:39:27.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:42.818 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-12-06T07:39:45.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-06T07:39:57.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:39:57.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:39:58.164 DEBUG:teuthology.orchestra.run.smithi084:> true 2023-12-06T07:39:58.500 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:39:58.500 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-390.el9.x86_64"... 2023-12-06T07:39:58.500 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2023-12-06T07:39:58.514 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-390.el9.x86_64 2023-12-06T07:39:58.518 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-06T07:39:58.518 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T07:39:58.518 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T07:39:59.518 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-06T07:39:59.519 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T07:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2023-12-06T07:39:59.615 DEBUG:teuthology.parallel:result is None 2023-12-06T07:40:00.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:40:00.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:40:01.130 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-12-06T07:40:01.488 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:40:01.488 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-390.el9.x86_64"... 2023-12-06T07:40:01.488 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-06T07:40:01.503 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-390.el9.x86_64 2023-12-06T07:40:01.503 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-06T07:40:01.503 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T07:40:01.503 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T07:40:02.504 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-06T07:40:02.504 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T07:40:02.571 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-12-06T07:40:02.609 DEBUG:teuthology.parallel:result is None 2023-12-06T07:40:02.610 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-06T07:40:02.617 INFO:teuthology.task.internal:Creating test directory... 2023-12-06T07:40:02.617 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T07:40:02.623 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T07:40:02.638 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-06T07:40:02.662 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-06T07:40:02.693 INFO:teuthology.task.internal:Creating archive directory... 2023-12-06T07:40:02.694 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T07:40:02.696 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T07:40:02.739 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-06T07:40:02.746 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-06T07:40:02.747 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T07:40:02.767 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T07:40:02.797 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T07:40:02.799 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T07:40:02.809 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T07:40:02.811 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T07:40:02.813 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-06T07:40:02.821 INFO:teuthology.task.internal:Configuring sudo... 2023-12-06T07:40:02.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T07:40:02.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T07:40:02.881 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-06T07:40:02.890 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-06T07:40:02.891 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T07:40:02.923 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T07:40:02.937 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T07:40:03.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T07:40:03.065 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T07:40:03.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T07:40:03.188 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:40:03.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T07:40:03.254 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T07:40:03.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T07:40:03.342 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T07:40:03.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T07:40:03.466 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:40:03.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T07:40:03.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2023-12-06T07:40:03.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-12-06T07:40:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T07:40:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T07:40:03.992 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-06T07:40:04.000 INFO:teuthology.task.internal:Starting timer... 2023-12-06T07:40:04.000 INFO:teuthology.run_tasks:Running task pcp... 2023-12-06T07:40:04.047 INFO:teuthology.run_tasks:Running task selinux... 2023-12-06T07:40:04.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2023-12-06T07:40:04.143 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2023-12-06T07:40:04.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2023-12-06T07:40:04.272 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2023-12-06T07:40:04.274 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-06T07:40:04.274 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2023-12-06T07:40:04.310 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2023-12-06T07:40:04.310 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2023-12-06T07:40:04.641 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2023-12-06T07:40:04.641 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2023-12-06T07:40:04.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T07:40:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T07:40:04.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T07:40:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T07:40:04.725 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-06T07:40:04.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2023-12-06T07:40:04.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2023-12-06T07:40:04.819 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-06T07:40:04.829 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-06T07:40:04.834 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-06T07:40:04.951 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-06T07:40:04.969 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-06T07:40:04.970 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-06T07:45:43.133 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2023-12-06T07:45:43.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:45:43.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:45:43.219 DEBUG:teuthology.orchestra.run.smithi084:> true 2023-12-06T07:45:43.288 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-12-06T07:45:43.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:45:43.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T07:45:43.371 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-12-06T07:45:43.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-06T07:45:43.437 INFO:teuthology.run_tasks:Running task clock... 2023-12-06T07:45:43.450 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-06T07:45:43.450 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T07:45:43.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T07:45:43.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T07:45:43.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T07:45:43.482 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T07:45:43.498 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T07:45:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T07:45:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T07:45:43.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2023-12-06T07:45:43.541 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2023-12-06T07:45:43.554 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2023-12-06T07:45:43.557 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T07:45:43.568 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2023-12-06T07:45:43.573 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T07:45:43.584 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T07:45:43.600 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T07:45:43.616 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.643 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T07:45:43.647 INFO:teuthology.run_tasks:Running task install... 2023-12-06T07:45:43.657 DEBUG:teuthology.task.install:project ceph 2023-12-06T07:45:43.657 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}} 2023-12-06T07:45:43.657 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'} 2023-12-06T07:45:43.657 INFO:teuthology.task.install:Using flavor: default 2023-12-06T07:45:43.663 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-06T07:45:43.663 INFO:teuthology.task.install:extra packages: [] 2023-12-06T07:45:43.664 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'tag': None, 'wait_for_package': False} 2023-12-06T07:45:43.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T07:45:43.666 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'tag': None, 'wait_for_package': False} 2023-12-06T07:45:43.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T07:45:43.812 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/ 2023-12-06T07:45:43.812 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7644.g96fa2578 2023-12-06T07:45:43.817 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/ 2023-12-06T07:45:43.817 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7644.g96fa2578 2023-12-06T07:45:43.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T07:45:43.920 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:45:43.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T07:45:43.926 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-12-04-1848/96fa2578c622ca9c169af6f426ca8b1616130e58/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T07:45:43.926 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:45:43.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T07:45:43.948 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-06T07:45:43.948 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96fa2578c622ca9c169af6f426ca8b1616130e58/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T07:45:43.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-06T07:45:43.953 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96fa2578c622ca9c169af6f426ca8b1616130e58/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T07:45:44.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T07:45:44.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T07:45:44.099 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T07:45:44.101 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T07:45:44.165 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2023-12-06T07:45:44.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2023-12-06T07:45:44.167 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2023-12-06T07:45:44.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2023-12-06T07:45:44.414 INFO:teuthology.orchestra.run.smithi084.stdout:73 files removed 2023-12-06T07:45:44.423 INFO:teuthology.orchestra.run.smithi107.stdout:73 files removed 2023-12-06T07:45:44.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2023-12-06T07:45:44.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2023-12-06T07:45:45.272 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 155 kB/s | 78 kB 00:00 2023-12-06T07:45:45.300 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 144 kB/s | 78 kB 00:00 2023-12-06T07:45:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-06T07:45:45.776 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2023-12-06T07:45:46.072 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2023-12-06T07:45:46.201 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2023-12-06T07:45:46.566 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-06T07:45:46.703 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-06T07:45:48.120 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2023-12-06T07:45:48.250 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2023-12-06T07:45:49.003 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-06T07:45:50.879 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 23 MB/s | 18 MB 00:00 2023-12-06T07:45:51.391 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-12-06T07:45:53.758 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 14 MB/s | 18 MB 00:01 2023-12-06T07:45:54.713 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-06T07:45:56.072 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2023-12-06T07:45:57.973 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-06T07:45:58.392 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.8 MB 00:01 2023-12-06T07:46:00.300 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2023-12-06T07:46:02.614 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-12-06T07:46:02.735 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 20 MB 00:02 2023-12-06T07:46:05.224 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:05.227 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:05.227 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:46:05.227 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:05.227 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 8.6 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.3 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.2 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 24 k 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-06T07:46:05.228 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 718 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 476 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.8 M 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 45 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 129 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 161 k 2023-12-06T07:46:05.229 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 321 k 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 297 k 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 99 k 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout:Install 27 Packages 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2023-12-06T07:46:05.230 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:05.231 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 55 M 2023-12-06T07:46:05.231 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:46:05.882 INFO:teuthology.orchestra.run.smithi084.stdout:(1/29): ceph-base-18.0.0-7644.g96fa2578.el9.x86 10 MB/s | 5.2 MB 00:00 2023-12-06T07:46:05.924 INFO:teuthology.orchestra.run.smithi084.stdout:(2/29): ceph-selinux-18.0.0-7644.g96fa2578.el9. 580 kB/s | 24 kB 00:00 2023-12-06T07:46:05.991 INFO:teuthology.orchestra.run.smithi084.stdout:(3/29): libcephfs2-18.0.0-7644.g96fa2578.el9.x8 11 MB/s | 718 kB 00:00 2023-12-06T07:46:06.049 INFO:teuthology.orchestra.run.smithi084.stdout:(4/29): libradosstriper1-18.0.0-7644.g96fa2578. 8.0 MB/s | 476 kB 00:00 2023-12-06T07:46:06.349 INFO:teuthology.orchestra.run.smithi084.stdout:(5/29): librgw2-18.0.0-7644.g96fa2578.el9.x86_6 16 MB/s | 4.8 MB 00:00 2023-12-06T07:46:06.400 INFO:teuthology.orchestra.run.smithi084.stdout:(6/29): python3-ceph-argparse-18.0.0-7644.g96fa 904 kB/s | 45 kB 00:00 2023-12-06T07:46:06.450 INFO:teuthology.orchestra.run.smithi084.stdout:(7/29): python3-ceph-common-18.0.0-7644.g96fa25 2.5 MB/s | 129 kB 00:00 2023-12-06T07:46:06.500 INFO:teuthology.orchestra.run.smithi084.stdout:(8/29): python3-cephfs-18.0.0-7644.g96fa2578.el 3.2 MB/s | 161 kB 00:00 2023-12-06T07:46:06.567 INFO:teuthology.orchestra.run.smithi084.stdout:(9/29): python3-rados-18.0.0-7644.g96fa2578.el9 4.7 MB/s | 321 kB 00:00 2023-12-06T07:46:06.625 INFO:teuthology.orchestra.run.smithi084.stdout:(10/29): python3-rbd-18.0.0-7644.g96fa2578.el9. 5.0 MB/s | 297 kB 00:00 2023-12-06T07:46:06.684 INFO:teuthology.orchestra.run.smithi084.stdout:(11/29): python3-rgw-18.0.0-7644.g96fa2578.el9. 1.7 MB/s | 99 kB 00:00 2023-12-06T07:46:06.818 INFO:teuthology.orchestra.run.smithi084.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 794 kB/s | 106 kB 00:00 2023-12-06T07:46:06.868 INFO:teuthology.orchestra.run.smithi084.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-06T07:46:06.910 INFO:teuthology.orchestra.run.smithi084.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-12-06T07:46:06.977 INFO:teuthology.orchestra.run.smithi084.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2023-12-06T07:46:07.019 INFO:teuthology.orchestra.run.smithi084.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2023-12-06T07:46:07.061 INFO:teuthology.orchestra.run.smithi084.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2023-12-06T07:46:07.370 INFO:teuthology.orchestra.run.smithi084.stdout:(18/29): ceph-radosgw-18.0.0-7644.g96fa2578.el9 4.3 MB/s | 8.6 MB 00:01 2023-12-06T07:46:07.434 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-06T07:46:07.787 INFO:teuthology.orchestra.run.smithi084.stdout:(19/29): ceph-common-18.0.0-7644.g96fa2578.el9. 7.9 MB/s | 19 MB 00:02 2023-12-06T07:46:07.954 INFO:teuthology.orchestra.run.smithi084.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 346 kB/s | 309 kB 00:00 2023-12-06T07:46:08.074 INFO:teuthology.orchestra.run.smithi084.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2023-12-06T07:46:08.179 INFO:teuthology.orchestra.run.smithi084.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2023-12-06T07:46:08.271 INFO:teuthology.orchestra.run.smithi084.stdout:(23/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 50 kB/s | 24 kB 00:00 2023-12-06T07:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.0 MB/s | 837 kB 00:00 2023-12-06T07:46:08.621 INFO:teuthology.orchestra.run.smithi084.stdout:(25/29): libarrow-9.0.0-9.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2023-12-06T07:46:08.647 INFO:teuthology.orchestra.run.smithi084.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 510 kB/s | 192 kB 00:00 2023-12-06T07:46:08.689 INFO:teuthology.orchestra.run.smithi084.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2023-12-06T07:46:08.989 INFO:teuthology.orchestra.run.smithi084.stdout:(28/29): librados2-18.0.0-7644.g96fa2578.el9.x8 9.0 MB/s | 3.3 MB 00:00 2023-12-06T07:46:09.039 INFO:teuthology.orchestra.run.smithi084.stdout:(29/29): librbd1-18.0.0-7644.g96fa2578.el9.x86_ 7.7 MB/s | 3.0 MB 00:00 2023-12-06T07:46:09.044 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:09.044 INFO:teuthology.orchestra.run.smithi084.stdout:Total 14 MB/s | 55 MB 00:03 2023-12-06T07:46:09.045 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:46:09.077 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:46:09.077 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:46:09.530 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:46:09.530 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:46:10.105 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:46:10.121 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:10.121 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 8.6 M 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.3 M 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.2 M 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 24 k 2023-12-06T07:46:10.122 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 718 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 476 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.8 M 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse 2023-12-06T07:46:10.123 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 45 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 129 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 161 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 321 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 297 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 99 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:10.124 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:46:10.125 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:10.125 INFO:teuthology.orchestra.run.smithi107.stdout:Install 27 Packages 2023-12-06T07:46:10.125 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2023-12-06T07:46:10.125 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:10.125 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 55 M 2023-12-06T07:46:10.126 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:46:10.886 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:46:10.969 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-06T07:46:10.977 INFO:teuthology.orchestra.run.smithi107.stdout:(1/29): ceph-base-18.0.0-7644.g96fa2578.el9.x86 10 MB/s | 5.2 MB 00:00 2023-12-06T07:46:10.999 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-06T07:46:11.019 INFO:teuthology.orchestra.run.smithi107.stdout:(2/29): ceph-selinux-18.0.0-7644.g96fa2578.el9. 575 kB/s | 24 kB 00:00 2023-12-06T07:46:11.058 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-06T07:46:11.094 INFO:teuthology.orchestra.run.smithi107.stdout:(3/29): libcephfs2-18.0.0-7644.g96fa2578.el9.x8 9.4 MB/s | 718 kB 00:00 2023-12-06T07:46:11.161 INFO:teuthology.orchestra.run.smithi107.stdout:(4/29): libradosstriper1-18.0.0-7644.g96fa2578. 7.0 MB/s | 476 kB 00:00 2023-12-06T07:46:11.420 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-06T07:46:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:11.558 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:11.651 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 6/31 2023-12-06T07:46:11.678 INFO:teuthology.orchestra.run.smithi107.stdout:(5/29): librgw2-18.0.0-7644.g96fa2578.el9.x86_6 9.2 MB/s | 4.8 MB 00:00 2023-12-06T07:46:11.683 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:11.728 INFO:teuthology.orchestra.run.smithi107.stdout:(6/29): python3-ceph-argparse-18.0.0-7644.g96fa 906 kB/s | 45 kB 00:00 2023-12-06T07:46:11.729 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:11.758 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-06T07:46:11.778 INFO:teuthology.orchestra.run.smithi107.stdout:(7/29): python3-ceph-common-18.0.0-7644.g96fa25 2.5 MB/s | 129 kB 00:00 2023-12-06T07:46:11.802 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 9/31 2023-12-06T07:46:11.845 INFO:teuthology.orchestra.run.smithi107.stdout:(8/29): python3-cephfs-18.0.0-7644.g96fa2578.el 2.4 MB/s | 161 kB 00:00 2023-12-06T07:46:11.868 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T07:46:11.894 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T07:46:11.903 INFO:teuthology.orchestra.run.smithi107.stdout:(9/29): python3-rados-18.0.0-7644.g96fa2578.el9 5.4 MB/s | 321 kB 00:00 2023-12-06T07:46:11.959 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T07:46:11.962 INFO:teuthology.orchestra.run.smithi107.stdout:(10/29): python3-rbd-18.0.0-7644.g96fa2578.el9. 5.0 MB/s | 297 kB 00:00 2023-12-06T07:46:12.013 INFO:teuthology.orchestra.run.smithi107.stdout:(11/29): python3-rgw-18.0.0-7644.g96fa2578.el9. 1.9 MB/s | 99 kB 00:00 2023-12-06T07:46:12.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-06T07:46:12.093 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-06T07:46:12.134 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-06T07:46:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 748 kB/s | 106 kB 00:00 2023-12-06T07:46:12.166 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-06T07:46:12.204 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-06T07:46:12.213 INFO:teuthology.orchestra.run.smithi107.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2023-12-06T07:46:12.236 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-06T07:46:12.255 INFO:teuthology.orchestra.run.smithi107.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-12-06T07:46:12.330 INFO:teuthology.orchestra.run.smithi107.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-06T07:46:12.381 INFO:teuthology.orchestra.run.smithi107.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-12-06T07:46:12.430 INFO:teuthology.orchestra.run.smithi107.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 840 kB/s | 41 kB 00:00 2023-12-06T07:46:12.539 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-06T07:46:12.566 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T07:46:12.659 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T07:46:12.687 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 20/31 2023-12-06T07:46:12.723 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-06T07:46:13.097 INFO:teuthology.orchestra.run.smithi107.stdout:(18/29): ceph-radosgw-18.0.0-7644.g96fa2578.el9 3.3 MB/s | 8.6 MB 00:02 2023-12-06T07:46:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 431 kB/s | 309 kB 00:00 2023-12-06T07:46:13.150 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-06T07:46:13.186 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T07:46:13.251 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T07:46:13.319 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 24/31 2023-12-06T07:46:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 25/31 2023-12-06T07:46:13.381 INFO:teuthology.orchestra.run.smithi107.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 104 kB/s | 24 kB 00:00 2023-12-06T07:46:13.615 INFO:teuthology.orchestra.run.smithi107.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 208 kB/s | 49 kB 00:00 2023-12-06T07:46:13.940 INFO:teuthology.orchestra.run.smithi107.stdout:(22/29): ceph-common-18.0.0-7644.g96fa2578.el9. 5.5 MB/s | 19 MB 00:03 2023-12-06T07:46:13.998 INFO:teuthology.orchestra.run.smithi107.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 176 kB/s | 67 kB 00:00 2023-12-06T07:46:14.840 INFO:teuthology.orchestra.run.smithi107.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 228 kB/s | 192 kB 00:00 2023-12-06T07:46:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:16.099 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:16.157 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T07:46:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T07:46:16.241 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-06T07:46:16.241 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-06T07:46:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:16.574 INFO:teuthology.orchestra.run.smithi107.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 318 kB/s | 837 kB 00:02 2023-12-06T07:46:16.866 INFO:teuthology.orchestra.run.smithi107.stdout:(26/29): librados2-18.0.0-7644.g96fa2578.el9.x8 11 MB/s | 3.3 MB 00:00 2023-12-06T07:46:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:(27/29): librbd1-18.0.0-7644.g96fa2578.el9.x86_ 17 MB/s | 3.0 MB 00:00 2023-12-06T07:46:18.399 INFO:teuthology.orchestra.run.smithi107.stdout:(28/29): thrift-0.15.0-2.el9.x86_64.rpm 455 kB/s | 1.6 MB 00:03 2023-12-06T07:46:19.266 INFO:teuthology.orchestra.run.smithi107.stdout:(29/29): libarrow-9.0.0-9.el9.x86_64.rpm 733 kB/s | 4.4 MB 00:06 2023-12-06T07:46:19.270 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:19.270 INFO:teuthology.orchestra.run.smithi107.stdout:Total 6.0 MB/s | 55 MB 00:09 2023-12-06T07:46:19.271 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:46:19.306 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:46:19.306 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:46:19.759 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:46:19.759 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:46:21.130 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:46:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-06T07:46:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-06T07:46:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-06T07:46:21.620 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-06T07:46:21.669 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:21.752 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:21.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 6/31 2023-12-06T07:46:21.870 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:21.916 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:21.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-06T07:46:22.055 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 9/31 2023-12-06T07:46:22.121 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T07:46:22.148 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T07:46:22.196 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 11/31 2023-12-06T07:46:22.301 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-06T07:46:22.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-06T07:46:22.371 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-06T07:46:22.403 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-06T07:46:22.432 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-06T07:46:22.472 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-06T07:46:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-06T07:46:22.793 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T07:46:22.889 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 19/31 2023-12-06T07:46:22.932 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 20/31 2023-12-06T07:46:22.975 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-06T07:46:23.400 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-06T07:46:23.429 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T07:46:23.500 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 23/31 2023-12-06T07:46:23.584 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 24/31 2023-12-06T07:46:23.615 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 25/31 2023-12-06T07:46:25.756 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:25.802 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:25.852 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:25.852 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2023-12-06T07:46:25.852 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2023-12-06T07:46:25.853 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 26/31 2023-12-06T07:46:26.380 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T07:46:26.468 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 27/31 2023-12-06T07:46:26.469 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-06T07:46:26.469 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-06T07:46:26.469 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:26.498 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:26.673 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T07:46:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T07:46:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T07:46:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 1/31 2023-12-06T07:46:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2/31 2023-12-06T07:46:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 3/31 2023-12-06T07:46:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 4/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 6/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 8/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 9/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 11/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 12/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 13/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-06T07:46:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-06T07:46:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 30/31 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-06T07:46:28.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-3.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:28.978 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:29.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2023-12-06T07:46:29.909 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:27 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:46:30.064 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 41 M 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 164 k 2023-12-06T07:46:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:30.066 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 42 M 2023-12-06T07:46:30.067 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 173 M 2023-12-06T07:46:30.067 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:46:30.269 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 923 kB/s | 186 kB 00:00 2023-12-06T07:46:30.319 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-06T07:46:30.378 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-06T07:46:30.428 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-06T07:46:30.495 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): libcephsqlite-18.0.0-7644.g96fa2578.el9. 383 kB/s | 164 kB 00:00 2023-12-06T07:46:32.945 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-18.0.0-7644.g96fa2578.el9.x86_ 14 MB/s | 41 MB 00:02 2023-12-06T07:46:32.948 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:32.948 INFO:teuthology.orchestra.run.smithi084.stdout:Total 15 MB/s | 42 MB 00:02 2023-12-06T07:46:32.949 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:46:32.968 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:46:32.968 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:46:33.327 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:46:33.327 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:46:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:46:33.938 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-06T07:46:33.990 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-06T07:46:34.019 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-06T07:46:34.062 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-06T07:46:34.090 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T07:46:35.944 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:35.944 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2023-12-06T07:46:35.944 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2023-12-06T07:46:35.944 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2023-12-06T07:46:35.945 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2023-12-06T07:46:35.945 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2023-12-06T07:46:35.945 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2023-12-06T07:46:35.945 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2023-12-06T07:46:35.945 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:36.761 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T07:46:36.785 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 29/31 2023-12-06T07:46:36.786 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:36.786 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-06T07:46:36.786 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T07:46:36.786 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-06T07:46:36.786 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:36.788 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T07:46:36.856 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-06T07:46:36.856 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 1/31 2023-12-06T07:46:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2/31 2023-12-06T07:46:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 3/31 2023-12-06T07:46:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 4/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 5/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_6 6/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 7/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9. 8/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x8 9/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 10/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 11/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 12/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 13/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-06T07:46:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 28/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-06T07:46:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 30/31 2023-12-06T07:46:37.960 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T07:46:38.029 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-06T07:46:38.994 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-06T07:46:38.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:38.996 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:46:39.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2023-12-06T07:46:39.944 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:32 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:46:40.094 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:46:40.094 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:46:40.094 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 41 M 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 164 k 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2023-12-06T07:46:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:40.096 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 42 M 2023-12-06T07:46:40.096 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 173 M 2023-12-06T07:46:40.096 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 1/6 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2/6 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-06T07:46:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-06T07:46:40.285 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 992 kB/s | 186 kB 00:00 2023-12-06T07:46:40.343 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2023-12-06T07:46:40.377 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): libcephsqlite-18.0.0-7644.g96fa2578.el9. 585 kB/s | 164 kB 00:00 2023-12-06T07:46:40.402 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-06T07:46:40.427 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: jq-1.6-16.el9.x86_64 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-06T07:46:40.564 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-06T07:46:40.565 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:40.565 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:40.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2023-12-06T07:46:41.456 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:39 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:46:41.604 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 6.6 k 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 2.1 M 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.5 M 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 243 k 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.4 M 2023-12-06T07:46:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 18 M 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-06T07:46:41.608 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-06T07:46:41.609 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2023-12-06T07:46:41.610 INFO:teuthology.orchestra.run.smithi084.stdout:Install 36 Packages 2023-12-06T07:46:41.611 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:41.611 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2023-12-06T07:46:41.611 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 111 M 2023-12-06T07:46:41.611 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:46:42.219 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-18.0.0-7644.g96fa2578.el9.x86_ 19 MB/s | 41 MB 00:02 2023-12-06T07:46:42.222 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:42.222 INFO:teuthology.orchestra.run.smithi107.stdout:Total 20 MB/s | 42 MB 00:02 2023-12-06T07:46:42.223 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:46:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:46:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:46:42.387 INFO:teuthology.orchestra.run.smithi084.stdout:(1/36): ceph-18.0.0-7644.g96fa2578.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2023-12-06T07:46:42.598 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:46:42.598 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:46:42.737 INFO:teuthology.orchestra.run.smithi084.stdout:(2/36): ceph-mon-18.0.0-7644.g96fa2578.el9.x86_ 13 MB/s | 4.4 MB 00:00 2023-12-06T07:46:43.079 INFO:teuthology.orchestra.run.smithi084.stdout:(3/36): ceph-mds-18.0.0-7644.g96fa2578.el9.x86_ 2.4 MB/s | 2.1 MB 00:00 2023-12-06T07:46:43.160 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:46:43.188 INFO:teuthology.orchestra.run.smithi084.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7644.g96fa 2.2 MB/s | 243 kB 00:00 2023-12-06T07:46:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-06T07:46:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-06T07:46:43.271 INFO:teuthology.orchestra.run.smithi084.stdout:(5/36): ceph-mgr-18.0.0-7644.g96fa2578.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-12-06T07:46:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-06T07:46:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-06T07:46:43.355 INFO:teuthology.orchestra.run.smithi084.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2023-12-06T07:46:43.372 INFO:teuthology.orchestra.run.smithi084.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-12-06T07:46:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T07:46:43.397 INFO:teuthology.orchestra.run.smithi084.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2023-12-06T07:46:43.423 INFO:teuthology.orchestra.run.smithi084.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2023-12-06T07:46:43.448 INFO:teuthology.orchestra.run.smithi084.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2023-12-06T07:46:43.473 INFO:teuthology.orchestra.run.smithi084.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-06T07:46:43.665 INFO:teuthology.orchestra.run.smithi084.stdout:(12/36): ceph-osd-18.0.0-7644.g96fa2578.el9.x86 19 MB/s | 18 MB 00:00 2023-12-06T07:46:43.698 INFO:teuthology.orchestra.run.smithi084.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 4.6 MB/s | 1.1 MB 00:00 2023-12-06T07:46:43.724 INFO:teuthology.orchestra.run.smithi084.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 411 kB/s | 103 kB 00:00 2023-12-06T07:46:43.749 INFO:teuthology.orchestra.run.smithi084.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2023-12-06T07:46:43.774 INFO:teuthology.orchestra.run.smithi084.stdout:(16/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-06T07:46:43.799 INFO:teuthology.orchestra.run.smithi084.stdout:(17/36): python3-urllib3-1.26.5-4.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2023-12-06T07:46:43.925 INFO:teuthology.orchestra.run.smithi084.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2023-12-06T07:46:43.983 INFO:teuthology.orchestra.run.smithi084.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 237 kB/s | 43 kB 00:00 2023-12-06T07:46:44.008 INFO:teuthology.orchestra.run.smithi084.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 723 kB/s | 151 kB 00:00 2023-12-06T07:46:44.050 INFO:teuthology.orchestra.run.smithi084.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 265 kB/s | 11 kB 00:00 2023-12-06T07:46:44.084 INFO:teuthology.orchestra.run.smithi084.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2023-12-06T07:46:44.126 INFO:teuthology.orchestra.run.smithi084.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2023-12-06T07:46:44.143 INFO:teuthology.orchestra.run.smithi084.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2023-12-06T07:46:44.160 INFO:teuthology.orchestra.run.smithi084.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2023-12-06T07:46:44.185 INFO:teuthology.orchestra.run.smithi084.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 661 kB/s | 172 kB 00:00 2023-12-06T07:46:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2023-12-06T07:46:44.227 INFO:teuthology.orchestra.run.smithi084.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 692 kB/s | 46 kB 00:00 2023-12-06T07:46:44.252 INFO:teuthology.orchestra.run.smithi084.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2023-12-06T07:46:44.269 INFO:teuthology.orchestra.run.smithi084.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2023-12-06T07:46:44.294 INFO:teuthology.orchestra.run.smithi084.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2023-12-06T07:46:44.320 INFO:teuthology.orchestra.run.smithi084.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2023-12-06T07:46:44.337 INFO:teuthology.orchestra.run.smithi084.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 531 kB/s | 36 kB 00:00 2023-12-06T07:46:44.387 INFO:teuthology.orchestra.run.smithi084.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2023-12-06T07:46:44.412 INFO:teuthology.orchestra.run.smithi084.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 264 kB/s | 20 kB 00:00 2023-12-06T07:46:44.462 INFO:teuthology.orchestra.run.smithi084.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2023-12-06T07:46:44.468 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:44.468 INFO:teuthology.orchestra.run.smithi084.stdout:Total 11 MB/s | 30 MB 00:02 2023-12-06T07:46:44.479 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:46:44.517 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:46:44.517 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:46:44.923 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:46:44.923 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:46:46.233 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:46:46.297 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-06T07:46:46.483 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-06T07:46:46.484 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-06T07:46:46.484 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-06T07:46:46.484 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-06T07:46:46.498 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-06T07:46:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-06T07:46:46.667 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-06T07:46:46.716 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-06T07:46:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-06T07:46:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-06T07:46:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-06T07:46:46.929 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-06T07:46:46.984 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-06T07:46:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-06T07:46:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 5/6 2023-12-06T07:46:47.226 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-06T07:46:47.255 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-06T07:46:47.291 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-06T07:46:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T07:46:47.321 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-06T07:46:47.396 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-06T07:46:47.431 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T07:46:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-06T07:46:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-06T07:46:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-06T07:46:47.566 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-06T07:46:47.653 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-06T07:46:47.688 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-06T07:46:47.760 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-06T07:46:47.894 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-06T07:46:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T07:46:48.198 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 31/36 2023-12-06T07:46:48.227 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T07:46:48.251 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 6/6 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 1/6 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2/6 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-06T07:46:49.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-06T07:46:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-06T07:46:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:49.755 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:46:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T07:46:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:50.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2023-12-06T07:46:50.320 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T07:46:50.343 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:50.554 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T07:46:50.577 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T07:46:50.577 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:50.577 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-06T07:46:50.577 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T07:46:50.578 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T07:46:50.578 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:50.611 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:43 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:46:50.642 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T07:46:50.770 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 6.6 k 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:46:50.773 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 2.1 M 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.5 M 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 243 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 4.4 M 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 18 M 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-06T07:46:50.774 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-06T07:46:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2023-12-06T07:46:50.776 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:50.777 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 30 M 2023-12-06T07:46:50.777 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 111 M 2023-12-06T07:46:50.777 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:46:51.281 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-18.0.0-7644.g96fa2578.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2023-12-06T07:46:51.640 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mon-18.0.0-7644.g96fa2578.el9.x86_ 12 MB/s | 4.4 MB 00:00 2023-12-06T07:46:51.873 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mds-18.0.0-7644.g96fa2578.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2023-12-06T07:46:51.991 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7644.g96fa 2.0 MB/s | 243 kB 00:00 2023-12-06T07:46:52.024 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): ceph-mgr-18.0.0-7644.g96fa2578.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2023-12-06T07:46:52.191 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 934 kB/s | 187 kB 00:00 2023-12-06T07:46:52.218 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 863 kB/s | 144 kB 00:00 2023-12-06T07:46:52.242 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2023-12-06T07:46:52.267 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2023-12-06T07:46:52.292 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2023-12-06T07:46:52.326 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-06T07:46:52.466 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T07:46:52.466 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 1/36 2023-12-06T07:46:52.466 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 3/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 4/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 5/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 6/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-06T07:46:52.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T07:46:52.468 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-06T07:46:52.469 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-06T07:46:52.469 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-06T07:46:52.469 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-06T07:46:52.469 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-06T07:46:52.509 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): ceph-osd-18.0.0-7644.g96fa2578.el9.x86 20 MB/s | 18 MB 00:00 2023-12-06T07:46:52.543 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 4.6 MB/s | 1.1 MB 00:00 2023-12-06T07:46:52.568 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 426 kB/s | 103 kB 00:00 2023-12-06T07:46:52.594 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2023-12-06T07:46:52.619 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-06T07:46:52.644 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): python3-urllib3-1.26.5-4.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2023-12-06T07:46:52.944 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2023-12-06T07:46:53.061 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 104 kB/s | 43 kB 00:00 2023-12-06T07:46:53.170 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 288 kB/s | 151 kB 00:00 2023-12-06T07:46:53.270 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-12-06T07:46:53.370 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2023-12-06T07:46:53.401 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2023-12-06T07:46:53.477 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 219 kB/s | 23 kB 00:00 2023-12-06T07:46:53.501 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 308 kB/s | 172 kB 00:00 2023-12-06T07:46:53.527 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2023-12-06T07:46:53.529 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-06T07:46:53.530 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-06T07:46:53.531 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:53.585 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-12-06T07:46:53.611 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2023-12-06T07:46:53.644 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 673 kB/s | 79 kB 00:00 2023-12-06T07:46:53.711 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2023-12-06T07:46:53.778 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 672 kB/s | 90 kB 00:00 2023-12-06T07:46:53.804 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2023-12-06T07:46:53.820 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2023-12-06T07:46:53.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2023-12-06T07:46:53.921 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2023-12-06T07:46:53.955 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2023-12-06T07:46:53.980 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2023-12-06T07:46:53.986 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:53.987 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.4 MB/s | 30 MB 00:03 2023-12-06T07:46:53.999 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:46:54.042 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:46:54.042 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:46:54.461 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:52 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:46:54.472 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:46:54.473 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:46:54.553 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:46:54.603 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:54.604 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:46:54.604 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:54.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2023-12-06T07:46:55.241 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:53 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:46:55.385 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:55.386 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:46:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 778 k 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:46:55.387 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:55.388 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 778 k 2023-12-06T07:46:55.388 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 790 k 2023-12-06T07:46:55.388 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:46:55.820 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-18.0.0-7644.g96fa2578.el9.noarch.rpm 1.8 MB/s | 778 kB 00:00 2023-12-06T07:46:55.820 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:55.820 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.8 MB/s | 778 kB 00:00 2023-12-06T07:46:55.821 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:46:55.821 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:46:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:46:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:46:55.831 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:46:55.831 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:46:55.891 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-06T07:46:55.904 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:46:55.934 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-06T07:46:55.961 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-06T07:46:56.030 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-06T07:46:56.067 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-06T07:46:56.100 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-06T07:46:56.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-06T07:46:56.279 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-06T07:46:56.320 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-06T07:46:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-06T07:46:56.432 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:46:56.446 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-06T07:46:56.485 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-06T07:46:56.523 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:46:56.534 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-06T07:46:56.597 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-06T07:46:56.788 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-06T07:46:56.839 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-06T07:46:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-06T07:46:56.913 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-06T07:46:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:46:56.935 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-06T07:46:57.003 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-06T07:46:57.038 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T07:46:57.070 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-06T07:46:57.111 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-06T07:46:57.146 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-06T07:46:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:57.203 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-06T07:46:57.320 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-06T07:46:57.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T07:46:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-06T07:46:57.530 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-06T07:46:57.646 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T07:46:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 31/36 2023-12-06T07:46:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T07:46:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 32/36 2023-12-06T07:46:57.858 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:57.858 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-06T07:46:57.859 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T07:46:57.859 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-06T07:46:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:57.960 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:55 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:46:58.110 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 141 k 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:58.111 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 141 k 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 400 k 2023-12-06T07:46:58.112 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:46:58.332 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-18.0.0-7644.g96fa25 642 kB/s | 141 kB 00:00 2023-12-06T07:46:58.333 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:46:58.333 INFO:teuthology.orchestra.run.smithi084.stdout:Total 638 kB/s | 141 kB 00:00 2023-12-06T07:46:58.333 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:46:58.341 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:46:58.341 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:46:58.396 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:46:58.396 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:46:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:46:58.617 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:46:58.641 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:46:58.641 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:58.641 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-06T07:46:58.641 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:59.419 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 33/36 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-06T07:46:59.446 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:46:59.624 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:46:59.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2023-12-06T07:46:59.895 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 34/36 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-06T07:46:59.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:00.120 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T07:47:00.143 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 35/36 2023-12-06T07:47:00.143 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:00.143 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-06T07:47:00.143 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T07:47:00.144 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-06T07:47:00.144 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T07:47:00.437 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:58 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:00.530 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:00.580 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:00.581 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:00.581 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:00.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T07:47:01.227 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:59 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:01.373 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 1.8 M 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 28 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 15 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout:Install 7 Packages 2023-12-06T07:47:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:01.376 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.1 M 2023-12-06T07:47:01.376 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 8.7 M 2023-12-06T07:47:01.376 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:01.777 INFO:teuthology.orchestra.run.smithi084.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7644.g96fa 59 kB/s | 15 kB 00:00 2023-12-06T07:47:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7644.g96f 100 kB/s | 28 kB 00:00 2023-12-06T07:47:01.886 INFO:teuthology.orchestra.run.smithi084.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7644.g96fa2578 4.8 MB/s | 1.8 MB 00:00 2023-12-06T07:47:01.994 INFO:teuthology.orchestra.run.smithi084.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 36/36 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 1/36 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2/36 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 3/36 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 4/36 2023-12-06T07:47:02.090 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 5/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9. 6/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-06T07:47:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-06T07:47:02.092 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-06T07:47:02.094 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-06T07:47:02.094 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-06T07:47:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-06T07:47:02.161 INFO:teuthology.orchestra.run.smithi084.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 112 kB/s | 31 kB 00:00 2023-12-06T07:47:02.187 INFO:teuthology.orchestra.run.smithi084.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 106 kB/s | 41 kB 00:00 2023-12-06T07:47:02.387 INFO:teuthology.orchestra.run.smithi084.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 480 kB/s | 188 kB 00:00 2023-12-06T07:47:02.389 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:02.389 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-12-06T07:47:02.389 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:02.397 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:02.398 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:02.462 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:02.462 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:02.762 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:02.822 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-06T07:47:02.855 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-06T07:47:02.901 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-06T07:47:02.934 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-06T07:47:02.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 5/7 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-06T07:47:03.022 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-06T07:47:03.023 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-06T07:47:03.024 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:03.025 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:03.200 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 6/7 2023-12-06T07:47:03.232 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T07:47:03.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 1/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 2/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 3/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-06T07:47:03.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-06T07:47:03.955 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:56 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:04.047 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:04.095 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-06T07:47:04.095 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:04.095 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:04.096 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:04.097 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:04.098 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:04.098 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:04.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2023-12-06T07:47:04.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T07:47:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:57 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:04.896 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:02 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:04.899 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:04.899 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 778 k 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:04.900 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:04.901 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 778 k 2023-12-06T07:47:04.901 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 790 k 2023-12-06T07:47:04.901 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:05.042 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:05.043 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 7.4 M 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-06T07:47:05.044 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2023-12-06T07:47:05.045 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:05.046 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2023-12-06T07:47:05.046 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 209 M 2023-12-06T07:47:05.046 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:05.277 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 195 kB/s | 32 kB 00:00 2023-12-06T07:47:05.318 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2023-12-06T07:47:05.344 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-18.0.0-7644.g96fa2578.el9.noarch.rpm 1.7 MB/s | 778 kB 00:00 2023-12-06T07:47:05.345 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:05.345 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.7 MB/s | 778 kB 00:00 2023-12-06T07:47:05.345 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:05.350 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:05.350 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:05.356 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:05.356 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:05.360 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 834 kB/s | 34 kB 00:00 2023-12-06T07:47:05.419 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.8 MB/s | 3.0 MB 00:00 2023-12-06T07:47:05.433 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:05.619 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-12-06T07:47:05.686 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2023-12-06T07:47:05.712 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-06T07:47:05.797 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:05.820 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): ceph-mgr-diskprediction-local-18.0.0-76 10 MB/s | 7.4 MB 00:00 2023-12-06T07:47:05.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:05.962 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2023-12-06T07:47:06.121 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2023-12-06T07:47:06.146 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 604 kB/s | 195 kB 00:00 2023-12-06T07:47:06.219 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:06.507 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:06.713 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 328 kB/s | 245 kB 00:00 2023-12-06T07:47:06.717 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:06.717 INFO:teuthology.orchestra.run.smithi084.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-06T07:47:06.732 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:06.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T07:47:06.758 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:06.758 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:07.134 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:07.135 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:07.328 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:00 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:07.475 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:07.475 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 141 k 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 141 k 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 400 k 2023-12-06T07:47:07.476 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:07.701 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-18.0.0-7644.g96fa25 631 kB/s | 141 kB 00:00 2023-12-06T07:47:07.702 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:07.702 INFO:teuthology.orchestra.run.smithi107.stdout:Total 627 kB/s | 141 kB 00:00 2023-12-06T07:47:07.702 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:07.709 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:07.709 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:07.765 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:07.765 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:07.911 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-06T07:47:07.918 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:07.941 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-06T07:47:07.968 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:47:07.992 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:47:07.992 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:07.993 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-06T07:47:07.993 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:08.043 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-06T07:47:08.078 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-06T07:47:08.254 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T07:47:08.285 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T07:47:08.370 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578. 1/1 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:09.001 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:09.015 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-06T07:47:09.130 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-06T07:47:09.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2023-12-06T07:47:09.826 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:02 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:09.920 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:09.969 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:09.970 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:09.970 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:10.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T07:47:10.327 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-06T07:47:10.634 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:03 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:10.780 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:10.780 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:10.780 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 1.8 M 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 28 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 15 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:Install 7 Packages 2023-12-06T07:47:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:10.782 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.1 M 2023-12-06T07:47:10.782 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 8.7 M 2023-12-06T07:47:10.782 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:11.197 INFO:teuthology.orchestra.run.smithi107.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7644.g96f 143 kB/s | 28 kB 00:00 2023-12-06T07:47:11.214 INFO:teuthology.orchestra.run.smithi107.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7644.g96fa 71 kB/s | 15 kB 00:00 2023-12-06T07:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-06T07:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T07:47:11.405 INFO:teuthology.orchestra.run.smithi107.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7644.g96fa2578 4.3 MB/s | 1.8 MB 00:00 2023-12-06T07:47:11.548 INFO:teuthology.orchestra.run.smithi107.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 26 kB/s | 9.0 kB 00:00 2023-12-06T07:47:11.673 INFO:teuthology.orchestra.run.smithi107.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 89 kB/s | 41 kB 00:00 2023-12-06T07:47:11.815 INFO:teuthology.orchestra.run.smithi107.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 75 kB/s | 31 kB 00:00 2023-12-06T07:47:11.924 INFO:teuthology.orchestra.run.smithi107.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2023-12-06T07:47:11.927 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:11.927 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.8 MB/s | 2.1 MB 00:01 2023-12-06T07:47:11.927 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:11.935 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:11.935 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:11.998 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:11.998 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:12.301 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:12.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-06T07:47:12.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-06T07:47:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-06T07:47:12.470 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-06T07:47:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 5/7 2023-12-06T07:47:12.722 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 6/7 2023-12-06T07:47:12.752 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T07:47:13.338 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 7/7 2023-12-06T07:47:13.338 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9. 1/7 2023-12-06T07:47:13.338 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarc 2/7 2023-12-06T07:47:13.339 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.n 3/7 2023-12-06T07:47:13.339 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-06T07:47:13.339 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-06T07:47:13.339 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-06T07:47:13.650 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-06T07:47:13.651 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-06T07:47:13.651 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:13.651 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:13.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 1/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T07:47:13.916 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-06T07:47:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-06T07:47:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-06T07:47:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-06T07:47:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-06T07:47:14.319 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-06T07:47:14.319 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:14.319 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:14.320 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:14.494 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:07 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:14.640 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 7.4 M 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:14.641 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2023-12-06T07:47:14.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:14.643 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2023-12-06T07:47:14.643 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 209 M 2023-12-06T07:47:14.643 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:14.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2023-12-06T07:47:14.847 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 194 kB/s | 32 kB 00:00 2023-12-06T07:47:14.881 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2023-12-06T07:47:14.923 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2023-12-06T07:47:14.973 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-12-06T07:47:15.182 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-12-06T07:47:15.257 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2023-12-06T07:47:15.274 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:13 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:15.282 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-06T07:47:15.421 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 50 k 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:15.423 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2023-12-06T07:47:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:15.424 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.0 M 2023-12-06T07:47:15.424 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2023-12-06T07:47:15.424 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:15.474 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2023-12-06T07:47:15.617 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 172 kB/s | 24 kB 00:00 2023-12-06T07:47:15.624 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2023-12-06T07:47:15.634 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2023-12-06T07:47:15.649 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-06T07:47:15.702 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2023-12-06T07:47:15.726 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2023-12-06T07:47:15.743 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): ceph-mgr-rook-18.0.0-7644.g96fa2578.el9 187 kB/s | 50 kB 00:00 2023-12-06T07:47:15.768 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2023-12-06T07:47:15.785 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 731 kB/s | 43 kB 00:00 2023-12-06T07:47:16.002 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 138 kB/s | 32 kB 00:00 2023-12-06T07:47:16.019 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2023-12-06T07:47:16.128 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2023-12-06T07:47:16.153 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 538 kB/s | 197 kB 00:00 2023-12-06T07:47:16.211 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2023-12-06T07:47:16.225 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): ceph-mgr-diskprediction-local-18.0.0-7 4.8 MB/s | 7.4 MB 00:01 2023-12-06T07:47:16.341 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 343 kB/s | 245 kB 00:00 2023-12-06T07:47:16.345 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:16.345 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-06T07:47:16.357 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:16.380 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:16.380 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:16.528 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-12-06T07:47:16.532 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:16.532 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-12-06T07:47:16.533 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:16.545 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:16.545 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:16.639 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:16.641 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:16.767 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:16.768 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:17.186 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:17.357 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-06T07:47:17.413 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-06T07:47:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:17.468 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-06T07:47:17.505 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-06T07:47:17.547 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-06T07:47:17.548 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-06T07:47:17.588 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-06T07:47:17.652 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-06T07:47:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-06T07:47:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-06T07:47:17.765 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-06T07:47:17.814 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-06T07:47:17.895 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T07:47:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T07:47:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-06T07:47:18.528 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-06T07:47:18.578 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-06T07:47:18.612 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-06T07:47:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-06T07:47:18.690 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-06T07:47:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T07:47:18.803 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 1/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-06T07:47:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-06T07:47:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-06T07:47:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-06T07:47:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-06T07:47:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-06T07:47:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-06T07:47:19.868 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-06T07:47:19.868 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:19.868 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-06T07:47:19.869 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:19.870 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:20.035 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-06T07:47:20.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T07:47:20.734 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:18 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:20.884 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:20.884 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 133 k 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:20.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:20.886 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:20.887 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:47:20.887 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2023-12-06T07:47:20.887 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:20.888 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.8 M 2023-12-06T07:47:20.888 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 33 M 2023-12-06T07:47:20.888 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:21.077 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-06T07:47:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T07:47:21.447 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2023-12-06T07:47:21.605 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-12-06T07:47:21.630 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7644.g96fa2578.e 337 kB/s | 133 kB 00:00 2023-12-06T07:47:21.931 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 177 kB/s | 58 kB 00:00 2023-12-06T07:47:21.948 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2023-12-06T07:47:22.123 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 811 kB/s | 548 kB 00:00 2023-12-06T07:47:22.124 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:22.125 INFO:teuthology.orchestra.run.smithi084.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2023-12-06T07:47:22.125 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:22.134 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:22.135 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:22.223 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:22.224 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:22.521 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:22.695 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-06T07:47:22.749 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-06T07:47:23.266 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-06T07:47:23.359 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-06T07:47:23.420 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-06T07:47:23.445 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T07:47:23.511 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 12/12 2023-12-06T07:47:23.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2 1/12 2023-12-06T07:47:23.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-06T07:47:23.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-06T07:47:23.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-06T07:47:23.960 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-06T07:47:23.960 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:23.960 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:23.960 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:23.960 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:23.961 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T07:47:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/6 2023-12-06T07:47:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-06T07:47:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-06T07:47:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-06T07:47:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-06T07:47:24.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-06T07:47:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:24.548 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:24.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2023-12-06T07:47:24.931 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:17 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:25.078 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 50 k 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-06T07:47:25.079 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2023-12-06T07:47:25.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:25.081 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.0 M 2023-12-06T07:47:25.081 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2023-12-06T07:47:25.081 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:25.340 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 116 kB/s | 24 kB 00:00 2023-12-06T07:47:25.365 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 77 kB/s | 18 kB 00:00 2023-12-06T07:47:25.385 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:23 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:25.466 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2023-12-06T07:47:25.491 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2023-12-06T07:47:25.532 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:25.533 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2023-12-06T07:47:25.534 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:25.534 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:25.534 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:25.534 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 860 k 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:25.535 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:25.536 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2023-12-06T07:47:25.536 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:25.536 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 939 k 2023-12-06T07:47:25.536 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.7 M 2023-12-06T07:47:25.536 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:25.558 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 643 kB/s | 43 kB 00:00 2023-12-06T07:47:25.584 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): ceph-mgr-rook-18.0.0-7644.g96fa2578.el9 111 kB/s | 50 kB 00:00 2023-12-06T07:47:25.724 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 416 kB/s | 79 kB 00:00 2023-12-06T07:47:26.167 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 25 kB/s | 15 kB 00:00 2023-12-06T07:47:26.192 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 49 kB/s | 32 kB 00:00 2023-12-06T07:47:26.393 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2023-12-06T07:47:26.460 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 225 kB/s | 197 kB 00:00 2023-12-06T07:47:26.474 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-18.0.0-7644.g96fa2578.el9.x86_ 916 kB/s | 860 kB 00:00 2023-12-06T07:47:26.475 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:26.475 INFO:teuthology.orchestra.run.smithi084.stdout:Total 997 kB/s | 939 kB 00:00 2023-12-06T07:47:26.475 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:26.496 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:26.496 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:26.501 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 824 kB/s | 90 kB 00:00 2023-12-06T07:47:26.572 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:26.572 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:26.760 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-12-06T07:47:26.764 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:26.764 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2023-12-06T07:47:26.765 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:26.778 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:26.778 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:26.875 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:26.876 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:26.880 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-06T07:47:26.957 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:47:27.404 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:27.564 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-06T07:47:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-06T07:47:27.664 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:47:27.664 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T07:47:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-06T07:47:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-06T07:47:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-06T07:47:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-06T07:47:27.955 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-06T07:47:27.989 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-06T07:47:27.989 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:27.989 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:27.990 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-06T07:47:27.990 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:27.990 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:28.005 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-06T07:47:28.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-volume 2023-12-06T07:47:28.723 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-06T07:47:28.778 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-06T07:47:28.811 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-06T07:47:28.850 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-06T07:47:28.873 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T07:47:28.878 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:26 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:29.022 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:29.022 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:29.022 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2023-12-06T07:47:29.022 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 262 k 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 262 k 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.2 M 2023-12-06T07:47:29.023 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:29.644 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-volume-18.0.0-7644.g96fa2578.el9.noarch.rp 422 kB/s | 262 kB 00:00 2023-12-06T07:47:29.645 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:29.645 INFO:teuthology.orchestra.run.smithi084.stdout:Total 421 kB/s | 262 kB 00:00 2023-12-06T07:47:29.645 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:29.653 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:29.653 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:29.687 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:29.687 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 13/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 1/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-06T07:47:29.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-06T07:47:29.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-06T07:47:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:29.987 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:29.987 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-06T07:47:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-06T07:47:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:30.117 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:30.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T07:47:30.973 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:23 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:30.996 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:31.120 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:31.120 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:31.120 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:31.120 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 133 k 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2023-12-06T07:47:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:31.122 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 6.8 M 2023-12-06T07:47:31.122 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2023-12-06T07:47:31.122 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:31.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2023-12-06T07:47:31.363 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.2 MB/s | 228 kB 00:00 2023-12-06T07:47:31.613 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2023-12-06T07:47:31.638 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7644.g96fa2578.e 283 kB/s | 133 kB 00:00 2023-12-06T07:47:31.789 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2023-12-06T07:47:31.822 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2023-12-06T07:47:31.847 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2023-12-06T07:47:31.852 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:31.852 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.3 MB/s | 6.8 MB 00:00 2023-12-06T07:47:31.852 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:31.861 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:31.861 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:31.891 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:29 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:31.949 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:31.949 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:32.038 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 126 k 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:32.039 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 126 k 2023-12-06T07:47:32.040 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2023-12-06T07:47:32.040 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-06T07:47:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-06T07:47:32.540 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-18.0.0-7644.g96fa2578.el9.x86_64 253 kB/s | 126 kB 00:00 2023-12-06T07:47:32.541 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:32.541 INFO:teuthology.orchestra.run.smithi084.stdout:Total 252 kB/s | 126 kB 00:00 2023-12-06T07:47:32.542 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:32.547 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:32.547 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:32.604 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:32.604 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:32.757 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:32.855 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:33.006 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-06T07:47:33.093 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-06T07:47:33.153 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-06T07:47:33.178 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T07:47:33.348 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:33.654 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:33.655 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:33.655 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:33.655 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:33.655 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:33.655 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:33.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2023-12-06T07:47:34.002 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 6/6 2023-12-06T07:47:34.002 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 1/6 2023-12-06T07:47:34.002 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-06T07:47:34.002 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-06T07:47:34.003 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-06T07:47:34.003 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-06T07:47:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-06T07:47:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:34.312 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:34.509 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:32 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:34.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2023-12-06T07:47:34.606 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:34.657 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:34.657 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:34.658 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:34.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2023-12-06T07:47:35.125 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:35.269 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 860 k 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2023-12-06T07:47:35.270 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:35.271 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 939 k 2023-12-06T07:47:35.271 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.7 M 2023-12-06T07:47:35.271 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:35.307 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:33 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 31 k 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 k 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 139 k 2023-12-06T07:47:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:35.458 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 79 kB 00:00 2023-12-06T07:47:35.618 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-18.0.0-7644.g96fa2578.el9.x86_6 190 kB/s | 31 kB 00:00 2023-12-06T07:47:35.619 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:35.619 INFO:teuthology.orchestra.run.smithi084.stdout:Total 188 kB/s | 31 kB 00:00 2023-12-06T07:47:35.619 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:35.622 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:35.623 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:35.652 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:35.653 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:36.016 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-18.0.0-7644.g96fa2578.el9.x86_ 1.1 MB/s | 860 kB 00:00 2023-12-06T07:47:36.017 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:36.018 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 939 kB 00:00 2023-12-06T07:47:36.018 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:36.039 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:36.039 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:36.114 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:36.115 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:36.291 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:36.424 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-06T07:47:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:47:36.545 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:36.545 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:36.545 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:36.546 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:36.546 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:36.546 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:36.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2023-12-06T07:47:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:47:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T07:47:37.296 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:35 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:37.394 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:37.444 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:37.445 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:37.445 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:37.504 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-06T07:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-06T07:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:37.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2023-12-06T07:47:37.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2023-12-06T07:47:38.100 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:36 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:38.196 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:38.246 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:38.247 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:38.247 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:38.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2023-12-06T07:47:38.341 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:31 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:38.485 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:18.0.0-7644.g96fa2578.el9 ceph-noarch 262 k 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 262 k 2023-12-06T07:47:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.2 M 2023-12-06T07:47:38.487 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:38.895 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:36 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:38.898 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-volume-18.0.0-7644.g96fa2578.el9.noarch.rp 637 kB/s | 262 kB 00:00 2023-12-06T07:47:38.899 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:38.899 INFO:teuthology.orchestra.run.smithi107.stdout:Total 635 kB/s | 262 kB 00:00 2023-12-06T07:47:38.899 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:38.907 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:38.907 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:38.952 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:38.952 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:38.988 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:39.038 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:39.039 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:39.039 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:39.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2023-12-06T07:47:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:39.232 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:39.232 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-06T07:47:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:39.685 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:37 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:39.778 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:39.829 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:39.830 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:39.830 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:39.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2023-12-06T07:47:40.259 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 1/1 2023-12-06T07:47:40.260 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:40.260 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:40.260 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:18.0.0-7644.g96fa2578.el9.noarch 2023-12-06T07:47:40.260 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:40.260 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:40.482 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:38 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:40.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2023-12-06T07:47:40.577 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:40.628 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:40.629 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:40.629 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:40.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2023-12-06T07:47:41.161 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:41.276 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:39 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:41.305 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:41.305 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 126 k 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:41.306 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:41.307 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 126 k 2023-12-06T07:47:41.307 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 456 k 2023-12-06T07:47:41.307 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:41.371 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:41.421 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:41.422 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:47:41.422 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:41.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2023-12-06T07:47:41.637 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-18.0.0-7644.g96fa2578.el9.x86_64 383 kB/s | 126 kB 00:00 2023-12-06T07:47:41.638 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:41.638 INFO:teuthology.orchestra.run.smithi107.stdout:Total 381 kB/s | 126 kB 00:00 2023-12-06T07:47:41.638 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:41.644 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:41.644 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:41.698 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:41.698 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:41.852 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:41.944 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:42.073 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:40 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 85 k 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 85 k 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 231 k 2023-12-06T07:47:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:42.546 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-18.0.0-7644.g96fa2578.el9.x86_64.rpm 262 kB/s | 85 kB 00:00 2023-12-06T07:47:42.547 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:42.547 INFO:teuthology.orchestra.run.smithi084.stdout:Total 261 kB/s | 85 kB 00:00 2023-12-06T07:47:42.547 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:42.553 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:42.553 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:42.607 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:42.607 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:42.727 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:42.727 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:42.764 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:42.868 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:42.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2023-12-06T07:47:43.255 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:43.582 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:36 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:43.583 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:43.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:43.583 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:43.585 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:43.586 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:43.586 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:43.681 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:43.732 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:43.733 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:43.733 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:43.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2023-12-06T07:47:43.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2023-12-06T07:47:44.377 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:42 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:44.390 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:44.526 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.0 M 2023-12-06T07:47:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 12 M 2023-12-06T07:47:44.528 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:44.535 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 31 k 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:44.536 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:44.537 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 31 k 2023-12-06T07:47:44.537 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 139 k 2023-12-06T07:47:44.537 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:44.791 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-18.0.0-7644.g96fa2578.el9.x86_6 122 kB/s | 31 kB 00:00 2023-12-06T07:47:44.792 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:44.792 INFO:teuthology.orchestra.run.smithi107.stdout:Total 122 kB/s | 31 kB 00:00 2023-12-06T07:47:44.792 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:44.796 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:44.796 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:44.826 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:44.826 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:44.911 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:45.022 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:45.173 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-18.0.0-7644.g96fa2578.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2023-12-06T07:47:45.173 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:45.174 INFO:teuthology.orchestra.run.smithi084.stdout:Total 4.6 MB/s | 3.0 MB 00:00 2023-12-06T07:47:45.174 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:45.183 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:45.184 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:45.256 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:45.256 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:45.403 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:45.666 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:45.724 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T07:47:45.748 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:45.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2023-12-06T07:47:46.438 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:39 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:46.533 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:46.583 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:46.584 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:46.584 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:46.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2023-12-06T07:47:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:46.867 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:46.868 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:47.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2023-12-06T07:47:47.255 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:47.350 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:47.400 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:47.401 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:47.401 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:47.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2023-12-06T07:47:47.700 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:45 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:47.847 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 171 k 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:47:47.848 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:47.849 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 171 k 2023-12-06T07:47:47.849 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2023-12-06T07:47:47.849 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:48.073 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:48.167 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:48.201 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-18.0.0-7644.g96fa2578.el9.x86_64.rpm 485 kB/s | 171 kB 00:00 2023-12-06T07:47:48.201 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:48.201 INFO:teuthology.orchestra.run.smithi084.stdout:Total 483 kB/s | 171 kB 00:00 2023-12-06T07:47:48.202 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:48.209 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:48.210 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:48.218 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:48.219 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:48.219 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:48.265 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:48.265 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:48.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2023-12-06T07:47:48.420 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:47:48.523 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:48.891 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:48.985 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:48.990 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:49.035 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:49.036 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:49.036 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:49.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2023-12-06T07:47:49.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:49.295 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:49.295 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:47:49.295 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:49.295 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:49.295 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:47:49.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base-debuginfo 2023-12-06T07:47:49.701 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:49.795 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:49.847 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:49.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2023-12-06T07:47:50.118 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:48 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:47:50.262 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 73 M 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 21 M 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:47:50.263 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:47:50.264 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2023-12-06T07:47:50.264 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:47:50.264 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 94 M 2023-12-06T07:47:50.264 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 508 M 2023-12-06T07:47:50.264 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:47:50.520 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:43 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:50.614 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:47:50.668 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:50.669 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:47:50.669 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:50.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2023-12-06T07:47:51.340 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:44 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:51.485 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:51.486 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:51.486 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2023-12-06T07:47:51.486 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:51.486 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:51.486 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 85 k 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 85 k 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 231 k 2023-12-06T07:47:51.487 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:51.894 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-18.0.0-7644.g96fa2578.el9.x86_64.rpm 210 kB/s | 85 kB 00:00 2023-12-06T07:47:51.895 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:51.895 INFO:teuthology.orchestra.run.smithi107.stdout:Total 209 kB/s | 85 kB 00:00 2023-12-06T07:47:51.895 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:51.902 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:51.902 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:51.958 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:51.958 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:52.897 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:52.898 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:52.898 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:52.898 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:52.898 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:52.898 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:53.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2023-12-06T07:47:53.729 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:46 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:53.878 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:53.878 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 3.0 M 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.0 M 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 12 M 2023-12-06T07:47:53.879 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:54.481 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-18.0.0-7644.g96fa2578.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-12-06T07:47:54.482 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:54.482 INFO:teuthology.orchestra.run.smithi107.stdout:Total 5.0 MB/s | 3.0 MB 00:00 2023-12-06T07:47:54.482 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:54.562 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:54.562 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:54.984 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:55.049 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-06T07:47:55.074 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:56.214 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:56.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2023-12-06T07:47:56.511 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): ceph-base-debuginfo-18.0.0-7644.g96fa257 12 MB/s | 73 MB 00:06 2023-12-06T07:47:57.043 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:50 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 171 k 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 171 k 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 494 k 2023-12-06T07:47:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:47:57.402 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-debuginfo-18.0.0-7644.g96fa2578.el9 2.9 MB/s | 21 MB 00:07 2023-12-06T07:47:57.404 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:57.405 INFO:teuthology.orchestra.run.smithi084.stdout:Total 13 MB/s | 94 MB 00:07 2023-12-06T07:47:57.405 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:47:57.410 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:47:57.410 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:47:57.410 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-18.0.0-7644.g96fa2578.el9.x86_64.rpm 772 kB/s | 171 kB 00:00 2023-12-06T07:47:57.411 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:47:57.411 INFO:teuthology.orchestra.run.smithi107.stdout:Total 768 kB/s | 171 kB 00:00 2023-12-06T07:47:57.411 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:47:57.419 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:47:57.419 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:47:57.474 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:47:57.475 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:47:57.629 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:47:57.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:58.015 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:47:58.016 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:47:58.186 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:58.463 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:47:58.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base-debuginfo 2023-12-06T07:47:59.268 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:52 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:47:59.417 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 73 M 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 21 M 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:59.418 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 94 M 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 508 M 2023-12-06T07:47:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:48:00.946 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:48:03.181 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): ceph-debuginfo-18.0.0-7644.g96fa2578.el9 5.5 MB/s | 21 MB 00:03 2023-12-06T07:48:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T07:48:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T07:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-base-debuginfo-18.0.0-7644.g96fa257 5.9 MB/s | 73 MB 00:12 2023-12-06T07:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.6 MB/s | 94 MB 00:12 2023-12-06T07:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:48:12.049 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T07:48:12.049 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/2 2023-12-06T07:48:12.311 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:48:12.311 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:48:12.319 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:48:12.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-common-debuginfo 2023-12-06T07:48:13.138 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:11 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:48:13.284 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 404 M 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 404 M 2023-12-06T07:48:13.285 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.6 G 2023-12-06T07:48:13.286 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:48:15.255 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:48:22.856 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/2 2023-12-06T07:48:22.937 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T07:48:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 2/2 2023-12-06T07:48:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/2 2023-12-06T07:48:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2/2 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:48:26.591 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:48:26.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-common-debuginfo 2023-12-06T07:48:27.430 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:20 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:48:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 404 M 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:48:27.576 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 404 M 2023-12-06T07:48:27.577 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.6 G 2023-12-06T07:48:27.577 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-common-debuginfo-18.0.0-7644.g96fa2578.el9 14 MB/s | 404 MB 00:28 2023-12-06T07:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout:Total 14 MB/s | 404 MB 00:28 2023-12-06T07:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:48:43.934 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:48:43.934 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:49:00.911 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-common-debuginfo-18.0.0-7644.g96fa2578.el9 12 MB/s | 404 MB 00:33 2023-12-06T07:49:00.911 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:49:00.912 INFO:teuthology.orchestra.run.smithi107.stdout:Total 12 MB/s | 404 MB 00:33 2023-12-06T07:49:00.912 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:49:00.916 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:49:00.916 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:49:03.326 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:49:03.327 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:49:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:49:24.758 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:33.244 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:33.498 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:49:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:49:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:33.499 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:49:33.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-06T07:49:34.300 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:32 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:49:34.446 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:49:34.446 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:49:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:49:34.446 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:49:34.446 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.4 M 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.4 M 2023-12-06T07:49:34.447 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 6.2 M 2023-12-06T07:49:34.448 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:49:34.912 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-76 3.0 MB/s | 1.4 MB 00:00 2023-12-06T07:49:34.913 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:49:34.913 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.0 MB/s | 1.4 MB 00:00 2023-12-06T07:49:34.913 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:49:34.916 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:49:34.916 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:49:34.957 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:49:34.957 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:49:35.203 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:49:35.293 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:35.983 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:35.983 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:35.983 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:49:35.983 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:49:35.984 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:35.984 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:49:36.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-06T07:49:36.732 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:03:34 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 172 M 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:49:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:49:36.881 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:49:36.881 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:49:36.881 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 172 M 2023-12-06T07:49:36.881 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 730 M 2023-12-06T07:49:36.881 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:49:44.226 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:49:44.345 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:44.780 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-radosgw-debuginfo-18.0.0-7644.g96fa2578.el 22 MB/s | 172 MB 00:07 2023-12-06T07:49:44.781 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:49:44.781 INFO:teuthology.orchestra.run.smithi084.stdout:Total 22 MB/s | 172 MB 00:07 2023-12-06T07:49:44.782 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:49:44.785 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:49:44.786 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:49:45.828 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:49:45.829 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:49:53.269 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:53.539 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:49:53.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-06T07:49:54.373 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:47 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:49:54.524 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:49:54.525 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:49:54.525 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.4 M 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.4 M 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 6.2 M 2023-12-06T07:49:54.526 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:49:55.856 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-76 1.0 MB/s | 1.4 MB 00:01 2023-12-06T07:49:55.857 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:49:55.857 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.0 MB/s | 1.4 MB 00:01 2023-12-06T07:49:55.857 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:49:55.861 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:49:55.863 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:49:55.902 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:49:55.902 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:49:56.151 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:49:56.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:56.710 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:56.981 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7644. 1/1 2023-12-06T07:49:56.982 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:56.982 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:49:56.982 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:49:56.982 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:56.982 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:49:57.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-06T07:49:57.744 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:50 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:49:57.890 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 172 M 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 172 M 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 730 M 2023-12-06T07:49:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:50:03.496 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:50:03.600 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:06.009 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-radosgw-debuginfo-18.0.0-7644.g96fa2578.el 21 MB/s | 172 MB 00:08 2023-12-06T07:50:06.009 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:50:06.010 INFO:teuthology.orchestra.run.smithi107.stdout:Total 21 MB/s | 172 MB 00:08 2023-12-06T07:50:06.010 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:50:06.014 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:50:06.014 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:50:07.057 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:50:07.057 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:50:08.036 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:50:08.310 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:50:08.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test-debuginfo 2023-12-06T07:50:09.087 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:04:07 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:50:09.233 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 622 M 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:50:09.234 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 622 M 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.8 G 2023-12-06T07:50:09.235 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:50:25.077 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:50:25.286 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x 1/1 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:50:30.198 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:50:30.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test-debuginfo 2023-12-06T07:50:30.978 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:23 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:50:31.119 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:50:31.120 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:50:31.120 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:50:31.120 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 622 M 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 622 M 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.8 G 2023-12-06T07:50:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:51:23.215 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-test-debuginfo-18.0.0-7644.g96fa2578.el9.x 12 MB/s | 622 MB 00:52 2023-12-06T07:51:23.216 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:51:23.216 INFO:teuthology.orchestra.run.smithi107.stdout:Total 12 MB/s | 622 MB 00:52 2023-12-06T07:51:23.216 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:51:23.221 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:51:23.221 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:51:23.447 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-test-debuginfo-18.0.0-7644.g96fa2578.el9.x 8.4 MB/s | 622 MB 01:14 2023-12-06T07:51:23.448 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:51:23.448 INFO:teuthology.orchestra.run.smithi084.stdout:Total 8.4 MB/s | 622 MB 01:14 2023-12-06T07:51:23.448 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:51:23.453 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:51:23.453 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:51:26.917 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:51:26.918 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:51:27.138 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:51:27.139 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:52:32.227 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:52:32.334 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:32.562 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:52:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:41.656 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:41.752 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:41.935 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:42.019 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:52:42.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base-debuginfo 2023-12-06T07:52:42.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base-debuginfo 2023-12-06T07:52:42.801 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:06:35 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:52:42.862 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:06:40 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:52:42.898 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:52:42.945 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:52:42.946 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-12-06T07:52:42.946 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:52:42.957 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 is already installed. 2023-12-06T07:52:43.008 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:52:43.009 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2023-12-06T07:52:43.009 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:52:43.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-debuginfo 2023-12-06T07:52:43.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-debuginfo 2023-12-06T07:52:43.597 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:06:36 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:52:43.648 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:06:41 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:52:43.742 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:43.743 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 19 M 2023-12-06T07:52:43.744 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 100 M 2023-12-06T07:52:43.744 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:52:43.793 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:52:43.794 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:43.794 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:52:43.794 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 19 M 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 19 M 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 100 M 2023-12-06T07:52:43.795 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:52:44.871 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mgr-debuginfo-18.0.0-7644.g96fa2578.el9.x8 17 MB/s | 19 MB 00:01 2023-12-06T07:52:44.872 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:52:44.872 INFO:teuthology.orchestra.run.smithi107.stdout:Total 17 MB/s | 19 MB 00:01 2023-12-06T07:52:44.873 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:52:44.876 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:52:44.876 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:52:45.020 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:52:45.020 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:52:45.685 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mgr-debuginfo-18.0.0-7644.g96fa2578.el9.x8 10 MB/s | 19 MB 00:01 2023-12-06T07:52:45.686 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:52:45.686 INFO:teuthology.orchestra.run.smithi084.stdout:Total 10 MB/s | 19 MB 00:01 2023-12-06T07:52:45.686 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:52:45.690 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:52:45.690 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:52:45.835 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:52:45.835 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:52:47.180 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:52:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:52:48.074 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:48.659 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:48.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:48.660 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:52:48.660 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:52:48.660 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:48.660 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:52:48.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mds-debuginfo 2023-12-06T07:52:49.097 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:49.366 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:52:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:06:42 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:52:49.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mds-debuginfo 2023-12-06T07:52:49.585 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:52:49.585 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:52:49.585 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 47 M 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 47 M 2023-12-06T07:52:49.587 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 168 M 2023-12-06T07:52:49.587 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:52:50.152 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:06:48 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:52:50.299 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:52:50.300 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:50.300 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:52:50.300 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:50.300 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 47 M 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 47 M 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 168 M 2023-12-06T07:52:50.301 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:52:52.448 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mds-debuginfo-18.0.0-7644.g96fa2578.el9.x8 16 MB/s | 47 MB 00:02 2023-12-06T07:52:52.449 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:52:52.449 INFO:teuthology.orchestra.run.smithi107.stdout:Total 16 MB/s | 47 MB 00:02 2023-12-06T07:52:52.450 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:52:52.454 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:52:52.454 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:52:52.761 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:52:52.761 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:52:55.146 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mds-debuginfo-18.0.0-7644.g96fa2578.el9.x8 9.6 MB/s | 47 MB 00:04 2023-12-06T07:52:55.146 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:52:55.146 INFO:teuthology.orchestra.run.smithi084.stdout:Total 9.6 MB/s | 47 MB 00:04 2023-12-06T07:52:55.146 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:52:55.150 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:52:55.150 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:52:55.457 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:52:55.458 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:52:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:52:57.554 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:58.902 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:59.163 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:52:59.163 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:59.163 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:52:59.163 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:52:59.164 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:52:59.164 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:52:59.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mon-debuginfo 2023-12-06T07:52:59.922 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:06:52 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:53:00.070 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:53:00.070 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:53:00.070 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:00.070 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 111 M 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:53:00.071 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:00.072 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 111 M 2023-12-06T07:53:00.072 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 507 M 2023-12-06T07:53:00.072 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:53:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:01.729 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:53:01.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mon-debuginfo 2023-12-06T07:53:02.510 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:07:00 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:53:02.655 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 111 M 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:02.656 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 111 M 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 507 M 2023-12-06T07:53:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:53:07.016 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mon-debuginfo-18.0.0-7644.g96fa2578.el9.x8 16 MB/s | 111 MB 00:06 2023-12-06T07:53:07.016 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:53:07.017 INFO:teuthology.orchestra.run.smithi107.stdout:Total 16 MB/s | 111 MB 00:06 2023-12-06T07:53:07.017 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:53:07.021 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:53:07.021 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:53:07.709 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:53:07.710 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:53:09.653 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-mon-debuginfo-18.0.0-7644.g96fa2578.el9.x8 16 MB/s | 111 MB 00:06 2023-12-06T07:53:09.654 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:53:09.654 INFO:teuthology.orchestra.run.smithi084.stdout:Total 16 MB/s | 111 MB 00:06 2023-12-06T07:53:09.655 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:53:09.659 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:53:09.659 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:53:10.345 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:53:10.345 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:53:19.384 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:53:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:22.170 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:53:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:22.792 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:23.050 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:23.050 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:23.050 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:53:23.051 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:53:23.051 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:23.051 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:53:23.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-osd-debuginfo 2023-12-06T07:53:23.832 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:07:16 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 311 M 2023-12-06T07:53:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 311 M 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.3 G 2023-12-06T07:53:23.979 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:53:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:25.747 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:53:25.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-osd-debuginfo 2023-12-06T07:53:26.519 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:07:24 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:53:26.664 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:53:26.664 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 311 M 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 311 M 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.3 G 2023-12-06T07:53:26.665 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:53:40.138 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-osd-debuginfo-18.0.0-7644.g96fa2578.el9.x8 19 MB/s | 311 MB 00:16 2023-12-06T07:53:40.139 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:53:40.139 INFO:teuthology.orchestra.run.smithi107.stdout:Total 19 MB/s | 311 MB 00:16 2023-12-06T07:53:40.140 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:53:40.144 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:53:40.144 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:53:42.005 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:53:42.005 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:53:51.290 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-osd-debuginfo-18.0.0-7644.g96fa2578.el9.x8 13 MB/s | 311 MB 00:24 2023-12-06T07:53:51.291 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:53:51.291 INFO:teuthology.orchestra.run.smithi084.stdout:Total 13 MB/s | 311 MB 00:24 2023-12-06T07:53:51.291 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:53:51.295 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:53:51.296 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:53:53.140 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:53:53.140 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:13.985 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:14.272 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:22.414 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:22.715 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:22.716 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:22.716 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:54:22.716 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:22.716 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:22.716 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:54:22.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse-debuginfo 2023-12-06T07:54:23.496 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:16 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:54:23.642 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:54:23.643 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:23.643 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:54:23.643 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:23.643 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.8 M 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.8 M 2023-12-06T07:54:23.644 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 28 M 2023-12-06T07:54:23.645 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:54:24.160 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x 11 MB/s | 5.8 MB 00:00 2023-12-06T07:54:24.160 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:24.161 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 5.8 MB 00:00 2023-12-06T07:54:24.161 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:54:24.164 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:54:24.164 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:54:24.230 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:54:24.230 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:54:24.935 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:24.971 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:54:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:25.173 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:25.599 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:25.870 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:54:26.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel-debuginfo 2023-12-06T07:54:26.632 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:19 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:54:26.778 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 108 k 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 108 k 2023-12-06T07:54:26.779 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 440 k 2023-12-06T07:54:26.780 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:54:27.037 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-debuginfo-18.0.0-7644.g96fa2578. 420 kB/s | 108 kB 00:00 2023-12-06T07:54:27.038 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:27.038 INFO:teuthology.orchestra.run.smithi107.stdout:Total 418 kB/s | 108 kB 00:00 2023-12-06T07:54:27.038 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:54:27.041 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:54:27.042 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:54:27.074 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:54:27.074 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:54:27.186 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:27.293 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:27.932 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:54:28.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2-debuginfo 2023-12-06T07:54:28.706 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:21 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:54:28.852 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.1 M 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:28.853 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.1 M 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2023-12-06T07:54:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:54:30.624 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs2-debuginfo-18.0.0-7644.g96fa2578.el9. 2.9 MB/s | 5.1 MB 00:01 2023-12-06T07:54:30.625 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:30.625 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.9 MB/s | 5.1 MB 00:01 2023-12-06T07:54:30.625 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:54:30.629 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:54:30.629 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:54:30.696 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:54:30.696 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:54:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:31.988 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:32.260 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:32.261 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:32.261 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:54:32.261 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:32.261 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:32.261 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:54:32.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2-debuginfo 2023-12-06T07:54:33.070 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:26 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:54:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:33.215 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:54:33.215 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:33.215 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 43 M 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:54:33.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:33.217 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 43 M 2023-12-06T07:54:33.217 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 248 M 2023-12-06T07:54:33.217 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:54:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:54:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:33.459 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:54:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:33.461 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:54:33.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse-debuginfo 2023-12-06T07:54:34.280 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:08:32 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:54:34.436 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:54:34.437 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:54:34.437 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.8 M 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.8 M 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 28 M 2023-12-06T07:54:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:54:36.710 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x 2.5 MB/s | 5.8 MB 00:02 2023-12-06T07:54:36.711 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:36.711 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.5 MB/s | 5.8 MB 00:02 2023-12-06T07:54:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:54:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:54:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:54:36.783 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:54:36.784 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:37.495 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:54:37.591 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:38.473 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:54:38.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel-debuginfo 2023-12-06T07:54:39.247 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:08:37 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:54:39.391 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 108 k 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:54:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:39.393 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 108 k 2023-12-06T07:54:39.393 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 440 k 2023-12-06T07:54:39.393 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:54:39.729 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-debuginfo-18.0.0-7644.g96fa2578. 322 kB/s | 108 kB 00:00 2023-12-06T07:54:39.730 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:39.730 INFO:teuthology.orchestra.run.smithi084.stdout:Total 320 kB/s | 108 kB 00:00 2023-12-06T07:54:39.730 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:54:39.734 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:54:39.734 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:54:39.767 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:54:39.767 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:39.885 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:54:39.978 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:40.387 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:54:40.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2-debuginfo 2023-12-06T07:54:41.438 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:08:39 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:54:41.582 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 5.1 M 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:41.583 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.1 M 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2023-12-06T07:54:41.584 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:54:41.892 INFO:teuthology.orchestra.run.smithi107.stdout:librados2-debuginfo-18.0.0-7644.g96fa2578.el9.x 5.0 MB/s | 43 MB 00:08 2023-12-06T07:54:41.892 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:41.893 INFO:teuthology.orchestra.run.smithi107.stdout:Total 5.0 MB/s | 43 MB 00:08 2023-12-06T07:54:41.893 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:54:41.897 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:54:41.897 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:54:42.182 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs2-debuginfo-18.0.0-7644.g96fa2578.el9. 8.6 MB/s | 5.1 MB 00:00 2023-12-06T07:54:42.183 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:42.183 INFO:teuthology.orchestra.run.smithi084.stdout:Total 8.6 MB/s | 5.1 MB 00:00 2023-12-06T07:54:42.183 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:54:42.187 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:54:42.187 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:54:42.193 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:54:42.193 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:54:42.252 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:54:42.252 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:42.891 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:54:42.980 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:43.533 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:43.821 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:54:43.822 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:43.822 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:54:43.822 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:43.822 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:43.822 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:54:44.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2-debuginfo 2023-12-06T07:54:44.615 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:08:42 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:54:44.762 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 43 M 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:54:44.763 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:44.764 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:54:44.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:44.764 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 43 M 2023-12-06T07:54:44.764 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 248 M 2023-12-06T07:54:44.764 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:54:47.076 INFO:teuthology.orchestra.run.smithi084.stdout:librados2-debuginfo-18.0.0-7644.g96fa2578.el9.x 19 MB/s | 43 MB 00:02 2023-12-06T07:54:47.077 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:47.077 INFO:teuthology.orchestra.run.smithi084.stdout:Total 19 MB/s | 43 MB 00:02 2023-12-06T07:54:47.078 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:54:47.082 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:54:47.082 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:54:47.160 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:47.377 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:54:47.378 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:49.125 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:49.398 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:54:49.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1-debuginfo 2023-12-06T07:54:50.195 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:43 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:50.340 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 50 M 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 294 M 2023-12-06T07:54:50.341 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:54:52.297 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:54:52.398 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:52.869 INFO:teuthology.orchestra.run.smithi107.stdout:librbd1-debuginfo-18.0.0-7644.g96fa2578.el9.x86 20 MB/s | 50 MB 00:02 2023-12-06T07:54:52.870 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:52.870 INFO:teuthology.orchestra.run.smithi107.stdout:Total 20 MB/s | 50 MB 00:02 2023-12-06T07:54:52.870 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:54:52.874 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:54:52.874 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:54:53.212 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:54:53.212 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:54:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:54.476 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_ 1/1 2023-12-06T07:54:54.477 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:54.477 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:54:54.477 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:54:54.477 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:54.477 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:54:54.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1-debuginfo 2023-12-06T07:54:55.293 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:08:53 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:54:55.436 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 50 M 2023-12-06T07:54:55.437 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 294 M 2023-12-06T07:54:55.438 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:54:58.061 INFO:teuthology.orchestra.run.smithi084.stdout:librbd1-debuginfo-18.0.0-7644.g96fa2578.el9.x86 19 MB/s | 50 MB 00:02 2023-12-06T07:54:58.062 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:54:58.062 INFO:teuthology.orchestra.run.smithi084.stdout:Total 19 MB/s | 50 MB 00:02 2023-12-06T07:54:58.062 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:54:58.067 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:54:58.067 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:54:58.403 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:54:58.403 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:54:58.978 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:54:59.094 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:01.254 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:01.538 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:01.539 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:01.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs-debuginfo 2023-12-06T07:55:02.310 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:55 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 615 k 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:02.455 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 615 k 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.1 M 2023-12-06T07:55:02.456 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:02.830 INFO:teuthology.orchestra.run.smithi107.stdout:python3-cephfs-debuginfo-18.0.0-7644.g96fa2578. 1.6 MB/s | 615 kB 00:00 2023-12-06T07:55:02.831 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:02.831 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.6 MB/s | 615 kB 00:00 2023-12-06T07:55:02.831 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:02.833 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:02.833 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:02.869 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:02.869 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:03.144 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:03.531 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:03.812 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:04.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados-debuginfo 2023-12-06T07:55:04.156 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:04.258 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:04.627 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:57 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 770 k 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:04.772 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 770 k 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.7 M 2023-12-06T07:55:04.773 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:05.066 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rados-debuginfo-18.0.0-7644.g96fa2578.e 2.6 MB/s | 770 kB 00:00 2023-12-06T07:55:05.066 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:05.067 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.6 MB/s | 770 kB 00:00 2023-12-06T07:55:05.067 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:05.070 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:05.070 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:05.107 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:05.107 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:05.284 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:05.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:05.779 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:06.048 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:06.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd-debuginfo 2023-12-06T07:55:06.410 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:06.682 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:06.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:06.683 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:06.683 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:06.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:06.683 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:06.844 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:08:59 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:06.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs-debuginfo 2023-12-06T07:55:06.991 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:06.991 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:06.991 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:06.991 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.1 M 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.1 M 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 4.3 M 2023-12-06T07:55:06.992 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:07.476 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:05 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:07.486 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rbd-debuginfo-18.0.0-7644.g96fa2578.el9 2.3 MB/s | 1.1 MB 00:00 2023-12-06T07:55:07.487 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:07.487 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.3 MB/s | 1.1 MB 00:00 2023-12-06T07:55:07.487 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:07.490 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:07.490 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:07.530 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:07.530 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:07.621 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 615 k 2023-12-06T07:55:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 615 k 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.1 M 2023-12-06T07:55:07.623 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:07.748 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:07.859 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:07.904 INFO:teuthology.orchestra.run.smithi084.stdout:python3-cephfs-debuginfo-18.0.0-7644.g96fa2578. 2.1 MB/s | 615 kB 00:00 2023-12-06T07:55:07.905 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:07.905 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.1 MB/s | 615 kB 00:00 2023-12-06T07:55:07.905 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:07.908 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:07.909 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:07.944 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:07.945 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:08.111 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:08.203 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:08.523 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:08.524 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:08.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw-debuginfo 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9 1/1 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:08.921 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:09.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados-debuginfo 2023-12-06T07:55:09.332 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:09:02 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:09.478 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 337 k 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 337 k 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.0 M 2023-12-06T07:55:09.479 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:09.682 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:07 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:09.764 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rgw-debuginfo-18.0.0-7644.g96fa2578.el9 1.2 MB/s | 337 kB 00:00 2023-12-06T07:55:09.765 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:09.765 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 337 kB 00:00 2023-12-06T07:55:09.765 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:09.768 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:09.768 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:09.802 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:09.803 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:09.826 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 770 k 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:09.827 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:09.828 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 770 k 2023-12-06T07:55:09.828 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.7 M 2023-12-06T07:55:09.828 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:09.941 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:10.053 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:10.124 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rados-debuginfo-18.0.0-7644.g96fa2578.e 2.5 MB/s | 770 kB 00:00 2023-12-06T07:55:10.124 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:10.125 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.5 MB/s | 770 kB 00:00 2023-12-06T07:55:10.125 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:10.128 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:10.128 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:10.165 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:10.165 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:10.342 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:10.417 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:10.456 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:10.681 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:10.682 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:10.877 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:10.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse-debuginfo 2023-12-06T07:55:11.154 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9. 1/1 2023-12-06T07:55:11.155 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:11.155 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:11.155 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:11.155 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:11.155 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:11.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd-debuginfo 2023-12-06T07:55:11.462 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:09:04 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:11.606 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:11.607 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:11.607 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 506 k 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 506 k 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.5 M 2023-12-06T07:55:11.608 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:11.934 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:09 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:12.034 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x8 1.2 MB/s | 506 kB 00:00 2023-12-06T07:55:12.035 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:12.035 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 506 kB 00:00 2023-12-06T07:55:12.035 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:12.038 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:12.038 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:12.074 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:12.074 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:12.078 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:12.079 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:12.079 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:12.079 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:12.079 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:12.079 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.1 M 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.1 M 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 4.3 M 2023-12-06T07:55:12.080 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:12.232 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:12.341 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:12.551 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rbd-debuginfo-18.0.0-7644.g96fa2578.el9 2.4 MB/s | 1.1 MB 00:00 2023-12-06T07:55:12.552 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:12.552 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2023-12-06T07:55:12.552 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:12.555 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:12.556 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:12.594 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:12.595 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:12.735 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:12.813 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:13.006 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:13.006 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:13.006 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:13.006 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:13.006 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:13.007 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:13.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror-debuginfo 2023-12-06T07:55:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:13.603 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:13.603 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:13.604 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:13.604 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:13.604 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:13.604 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:13.815 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:09:06 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:13.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw-debuginfo 2023-12-06T07:55:13.961 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 50 M 2023-12-06T07:55:13.962 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 284 M 2023-12-06T07:55:13.963 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:14.398 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:12 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:14.545 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 337 k 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 337 k 2023-12-06T07:55:14.933 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.0 M 2023-12-06T07:55:14.934 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:15.221 INFO:teuthology.orchestra.run.smithi084.stdout:python3-rgw-debuginfo-18.0.0-7644.g96fa2578.el9 500 kB/s | 337 kB 00:00 2023-12-06T07:55:15.222 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:15.222 INFO:teuthology.orchestra.run.smithi084.stdout:Total 499 kB/s | 337 kB 00:00 2023-12-06T07:55:15.222 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:15.225 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:15.225 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:15.261 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:15.261 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:15.407 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:15.915 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:16.188 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x8 1/1 2023-12-06T07:55:16.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:16.189 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:16.189 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:16.189 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:16.189 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:16.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse-debuginfo 2023-12-06T07:55:16.980 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:14 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:17.127 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:17.127 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:17.127 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:17.127 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:17.127 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 506 k 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 506 k 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.5 M 2023-12-06T07:55:17.128 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:17.615 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-debuginfo-18.0.0-7644.g96fa2578.el9.x8 1.0 MB/s | 506 kB 00:00 2023-12-06T07:55:17.616 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:17.616 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.0 MB/s | 506 kB 00:00 2023-12-06T07:55:17.616 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:17.619 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:17.619 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:17.645 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-debuginfo-18.0.0-7644.g96fa2578.el9. 14 MB/s | 50 MB 00:03 2023-12-06T07:55:17.645 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:17.646 INFO:teuthology.orchestra.run.smithi107.stdout:Total 13 MB/s | 50 MB 00:03 2023-12-06T07:55:17.646 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:17.650 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:17.651 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:17.655 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:17.655 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:17.813 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:17.922 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:17.975 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:17.975 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:18.351 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:18.627 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_6 1/1 2023-12-06T07:55:18.627 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:18.627 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:18.628 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:18.628 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:18.628 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:18.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror-debuginfo 2023-12-06T07:55:19.420 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:17 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:19.566 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 50 M 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 50 M 2023-12-06T07:55:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 284 M 2023-12-06T07:55:19.568 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:21.943 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-debuginfo-18.0.0-7644.g96fa2578.el9. 21 MB/s | 50 MB 00:02 2023-12-06T07:55:21.944 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:21.945 INFO:teuthology.orchestra.run.smithi084.stdout:Total 21 MB/s | 50 MB 00:02 2023-12-06T07:55:21.945 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:21.949 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:21.949 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:22.273 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:22.273 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:23.614 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:23.725 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:25.738 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:26.013 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:26.014 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:26.014 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:26.014 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:26.014 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:26.014 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:26.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd-debuginfo 2023-12-06T07:55:26.837 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:09:19 ago on Wed 06 Dec 2023 07:46:07 AM UTC. 2023-12-06T07:55:26.983 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.2 M 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.2 M 2023-12-06T07:55:26.984 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 5.4 M 2023-12-06T07:55:26.985 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-06T07:55:27.330 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-debuginfo-18.0.0-7644.g96fa2578.el9.x86 3.6 MB/s | 1.2 MB 00:00 2023-12-06T07:55:27.330 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:27.331 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.5 MB/s | 1.2 MB 00:00 2023-12-06T07:55:27.331 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-06T07:55:27.334 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-06T07:55:27.334 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-06T07:55:27.375 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-06T07:55:27.375 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-06T07:55:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-06T07:55:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:28.023 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:28.130 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:28.388 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-06T07:55:28.580 DEBUG:teuthology.parallel:result is None 2023-12-06T07:55:30.046 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:30.327 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86 1/1 2023-12-06T07:55:30.327 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:30.327 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:30.327 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:30.327 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:30.328 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:30.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd-debuginfo 2023-12-06T07:55:31.113 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:09:29 ago on Wed 06 Dec 2023 07:46:02 AM UTC. 2023-12-06T07:55:31.257 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2023-12-06T07:55:31.257 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7644.g96fa2578.el9 ceph 1.2 M 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.2 M 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 5.4 M 2023-12-06T07:55:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2023-12-06T07:55:31.569 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-debuginfo-18.0.0-7644.g96fa2578.el9.x86 4.0 MB/s | 1.2 MB 00:00 2023-12-06T07:55:31.569 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2023-12-06T07:55:31.570 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-12-06T07:55:31.570 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2023-12-06T07:55:31.573 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2023-12-06T07:55:31.573 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2023-12-06T07:55:31.613 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2023-12-06T07:55:31.613 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2023-12-06T07:55:31.842 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2023-12-06T07:55:31.942 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 1/1 2023-12-06T07:55:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:32.636 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2023-12-06T07:55:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-debuginfo-2:18.0.0-7644.g96fa2578.el9.x86_64 2023-12-06T07:55:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:32.636 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2023-12-06T07:55:32.839 DEBUG:teuthology.parallel:result is None 2023-12-06T07:55:32.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T07:55:33.003 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T07:55:33.023 INFO:teuthology.orchestra.run.smithi084.stdout:18.0.0-7644.g96fa2578.el9 2023-12-06T07:55:33.023 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7644.g96fa2578.el9 2023-12-06T07:55:33.024 INFO:teuthology.task.install:The correct ceph version 18.0.0-7644.g96fa2578 is installed. 2023-12-06T07:55:33.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96fa2578c622ca9c169af6f426ca8b1616130e58 2023-12-06T07:55:33.237 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T07:55:33.262 INFO:teuthology.orchestra.run.smithi107.stdout:18.0.0-7644.g96fa2578.el9 2023-12-06T07:55:33.262 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7644.g96fa2578.el9 2023-12-06T07:55:33.262 INFO:teuthology.task.install:The correct ceph version 18.0.0-7644.g96fa2578 is installed. 2023-12-06T07:55:33.262 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-06T07:55:33.263 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:33.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T07:55:33.289 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:33.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T07:55:33.334 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-06T07:55:33.334 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:33.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T07:55:33.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T07:55:33.426 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:33.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T07:55:33.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T07:55:33.526 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-06T07:55:33.527 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:33.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T07:55:33.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T07:55:33.619 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:33.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T07:55:33.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T07:55:33.719 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-06T07:55:33.719 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:33.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T07:55:33.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T07:55:33.812 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:33.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T07:55:33.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T07:55:33.911 INFO:teuthology.run_tasks:Running task ceph... 2023-12-06T07:55:33.977 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-06T07:55:33.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2023-12-06T07:55:33.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2023-12-06T07:55:34.008 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-06T07:55:34.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T07:55:34.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T07:55:34.076 INFO:tasks.ceph:Creating extra log directories... 2023-12-06T07:55:34.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T07:55:34.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T07:55:34.154 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-06T07:55:34.154 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-06T07:55:34.154 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417', 'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7479417', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sseshasa-testing-2023-12-04-1848'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'owner': 'scheduled_sseshasa@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5341, 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'sleep_before_teardown': 0, 'subset': '222/150000', 'suite': 'rados', 'suite_branch': 'wip-sseshasa-testing-2023-12-04-1848', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'targets': {'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQVssVISaQqOamGjBZt/mY8Q2tIklX0RUy5g1WQZzGogN+AXO2/DfEhuPAJ7R8ULDgF4tL4li1qOe2kmKwf8BQ=', 'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvIqCUOjcCj8G+G4cMbYPHfTHYBX87fiDKJMbdetC3yteJVYM9dkUHa/psN7FRl95bRJ+DNpXqfLM5Zv/zGt+M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'mon_recovery': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-06_06:16:39', 'tube': 'smithi', 'user': 'sseshasa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-06T07:55:34.155 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-06T07:55:34.179 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-06T07:55:34.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T07:55:34.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T07:55:34.267 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:34.267 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T07:55:34.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T07:55:34.314 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2023-12-06T07:55:34.368 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2023-12-06 07:47:45.972201290 +0000 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2023-12-06 07:45:13.522688808 +0000 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2023-12-06 07:45:13.522688808 +0000 2023-12-06T07:55:34.369 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2023-12-06 07:45:13.522688808 +0000 2023-12-06T07:55:34.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T07:55:34.434 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2023-12-06T07:55:34.434 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2023-12-06T07:55:34.435 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.00019536 s, 2.6 MB/s 2023-12-06T07:55:34.435 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T07:55:34.492 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2023-12-06T07:55:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2023-12-06 07:47:45.972201290 +0000 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2023-12-06 07:45:14.022679219 +0000 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2023-12-06 07:45:14.022679219 +0000 2023-12-06T07:55:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2023-12-06 07:45:14.022679219 +0000 2023-12-06T07:55:34.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T07:55:34.614 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2023-12-06T07:55:34.614 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2023-12-06T07:55:34.614 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000184575 s, 2.8 MB/s 2023-12-06T07:55:34.616 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T07:55:34.672 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2023-12-06 07:47:45.972201290 +0000 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2023-12-06 07:45:14.522669631 +0000 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2023-12-06 07:45:14.522669631 +0000 2023-12-06T07:55:34.727 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2023-12-06 07:45:14.522669631 +0000 2023-12-06T07:55:34.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T07:55:34.793 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2023-12-06T07:55:34.793 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2023-12-06T07:55:34.793 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000191082 s, 2.7 MB/s 2023-12-06T07:55:34.794 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T07:55:34.850 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2023-12-06T07:55:34.904 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2023-12-06 07:47:45.973201271 +0000 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2023-12-06 07:45:15.037659754 +0000 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2023-12-06 07:45:15.037659754 +0000 2023-12-06T07:55:34.905 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2023-12-06 07:45:15.037659754 +0000 2023-12-06T07:55:34.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T07:55:34.970 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2023-12-06T07:55:34.970 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2023-12-06T07:55:34.970 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000212754 s, 2.4 MB/s 2023-12-06T07:55:34.971 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T07:55:35.027 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:35.027 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:35.027 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T07:55:35.044 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T07:55:35.045 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-12-06T07:55:35.101 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T07:55:35.101 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:35.101 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-06T07:55:35.101 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:35.101 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:35.102 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-06 07:47:55.300034289 +0000 2023-12-06T07:55:35.102 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-06 07:45:13.516055562 +0000 2023-12-06T07:55:35.102 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-06 07:45:13.516055562 +0000 2023-12-06T07:55:35.102 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-12-06 07:45:13.516055562 +0000 2023-12-06T07:55:35.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T07:55:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-06T07:55:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-06T07:55:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000212206 s, 2.4 MB/s 2023-12-06T07:55:35.170 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T07:55:35.227 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:35.283 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-06 07:47:55.300034289 +0000 2023-12-06T07:55:35.284 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-06 07:45:14.006047310 +0000 2023-12-06T07:55:35.284 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-06 07:45:14.006047310 +0000 2023-12-06T07:55:35.284 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-12-06 07:45:14.006047310 +0000 2023-12-06T07:55:35.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T07:55:35.349 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-06T07:55:35.349 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-06T07:55:35.349 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00021577 s, 2.4 MB/s 2023-12-06T07:55:35.351 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T07:55:35.406 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-06 07:47:55.301034272 +0000 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-06 07:45:14.512038794 +0000 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-06 07:45:14.512038794 +0000 2023-12-06T07:55:35.461 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-12-06 07:45:14.512038794 +0000 2023-12-06T07:55:35.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T07:55:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-06T07:55:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-06T07:55:35.527 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000192676 s, 2.7 MB/s 2023-12-06T07:55:35.528 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T07:55:35.584 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-12-06 07:47:55.301034272 +0000 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-12-06 07:45:15.016030312 +0000 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-12-06 07:45:15.016030312 +0000 2023-12-06T07:55:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2023-12-06 07:45:15.016030312 +0000 2023-12-06T07:55:35.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T07:55:35.705 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-12-06T07:55:35.705 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-12-06T07:55:35.705 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000219919 s, 2.3 MB/s 2023-12-06T07:55:35.706 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T07:55:35.762 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:35.762 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-06T07:55:35.762 INFO:tasks.ceph:Generating config... 2023-12-06T07:55:35.763 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-06T07:55:35.763 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-12-06T07:55:35.763 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-12-06T07:55:35.763 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-12-06T07:55:35.764 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-06T07:55:35.764 INFO:tasks.ceph:[global] ms type = async 2023-12-06T07:55:35.764 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-06T07:55:35.764 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-06T07:55:35.764 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-12-06T07:55:35.765 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-06T07:55:35.766 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-06T07:55:35.766 INFO:tasks.ceph:Setting up mon.a... 2023-12-06T07:55:35.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-06T07:55:35.822 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.keyring 2023-12-06T07:55:35.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-06T07:55:35.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T07:55:35.936 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.84'), ('mon.c', '[v2:172.21.15.84:3301,v1:172.21.15.84:6790]'), ('mon.b', '172.21.15.107')] 2023-12-06T07:55:35.936 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790],172.21.15.107', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-06T07:55:35.937 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:35.937 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T07:55:35.991 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.84 --addv c '[v2:172.21.15.84:3301,v1:172.21.15.84:6790]' --add b 172.21.15.107 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: generated fsid aba07bce-2203-46e5-bd3f-65b300a2f9db 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = pacific 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 0 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:fsid aba07bce-2203-46e5-bd3f-65b300a2f9db 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:last_changed 2023-12-06T07:55:36.076779+0000 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:created 2023-12-06T07:55:36.076779+0000 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:min_mon_release 16 (pacific) 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:election_strategy: 3 2023-12-06T07:55:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2023-12-06T07:55:36.077 INFO:teuthology.orchestra.run.smithi084.stdout:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2023-12-06T07:55:36.077 INFO:teuthology.orchestra.run.smithi084.stdout:2: [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon.c 2023-12-06T07:55:36.077 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-06T07:55:36.078 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T07:55:36.092 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID aba07bce-2203-46e5-bd3f-65b300a2f9db... 2023-12-06T07:55:36.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-06T07:55:36.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2023-12-06T07:55:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2023-12-06T07:55:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = "172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790],172.21.15.107" 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: mon client directed command retry = 5 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: mon election default strategy = 3 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: ms bind msgr1 = False 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: ms bind msgr2 = True 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: ms inject socket failures = 5000 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: ms type = async 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: osd heartbeat grace = 80 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = aba07bce-2203-46e5-bd3f-65b300a2f9db 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.186 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2023-12-06T07:55:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: bdev async discard = True 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: bdev enable discard = True 2023-12-06T07:55:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore block size = 96636764160 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore fsck on mount = True 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore zero block detection = True 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluefs = 1 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluestore = 1 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: debug rocksdb = 10 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd full ratio = 0.9 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd nearfull ratio = 0.8 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: osd blocked scrub grace period = 3600 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify cached snaps = True 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify missing on start = True 2023-12-06T07:55:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: osd failsafe full ratio = 0.95 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd fast shutdown = False 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd max markdown count = 1000 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock override recovery settings = True 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd memory target = 1610612736 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd objectstore = bluestore 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue = debug_random 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue cut off = debug_random 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: osd shutdown pgref assert = True 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2023-12-06T07:55:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T07:55:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon min osdmap epochs = 50 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd crush smoke test = False 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune interval = 2 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune min = 15 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune txsize = 2 2023-12-06T07:55:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: mon scrub interval = 300 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: paxos service trim min = 10 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.c] 2023-12-06T07:55:36.193 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.b] 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2023-12-06T07:55:36.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2023-12-06T07:55:36.195 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2023-12-06T07:55:36.196 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790],172.21.15.107" 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: mon election default strategy = 3 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr1 = False 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr2 = True 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: ms type = async 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: osd heartbeat grace = 80 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = aba07bce-2203-46e5-bd3f-65b300a2f9db 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.197 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.198 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore zero block detection = True 2023-12-06T07:55:36.199 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd fast shutdown = False 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock override recovery settings = True 2023-12-06T07:55:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: osd memory target = 1610612736 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2023-12-06T07:55:36.201 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T07:55:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon min osdmap epochs = 50 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd crush smoke test = False 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune interval = 2 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune min = 15 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune txsize = 2 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub interval = 300 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: paxos service trim min = 10 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2023-12-06T07:55:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2023-12-06T07:55:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2023-12-06T07:55:36.205 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-06T07:55:36.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-06T07:55:36.301 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-06T07:55:36.301 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:36.301 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-06T07:55:36.316 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:36.316 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-06T07:55:36.370 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:55:36.370 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:36.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-06T07:55:36.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T07:55:36.449 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:55:36.449 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:36.503 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:55:36.504 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:36.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-06T07:55:36.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-06T07:55:36.543 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:55:36.543 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:36.597 INFO:tasks.ceph:Setting up mon nodes... 2023-12-06T07:55:36.597 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-06T07:55:36.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-06T07:55:36.651 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-06T07:55:36.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-06T07:55:36.711 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-06T07:55:36.714 INFO:tasks.ceph:Setting up mds nodes... 2023-12-06T07:55:36.714 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-06T07:55:36.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-06T07:55:36.756 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-06T07:55:36.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-06T07:55:36.813 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-06T07:55:36.828 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-06T07:55:36.829 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-06T07:55:36.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-06T07:55:36.855 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:36.855 INFO:tasks.ceph:role: osd.0 2023-12-06T07:55:36.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:55:36.855 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:36.929 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:37.427 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2023-12-06T07:55:37.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2023-12-06T07:55:37.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-06T07:55:37.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-06T07:55:37.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-06T07:55:37.642 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:37.643 INFO:tasks.ceph:role: osd.1 2023-12-06T07:55:37.643 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:55:37.643 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T07:55:37.717 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:37.717 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:37.717 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:37.717 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:37.717 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:37.718 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:37.718 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:37.718 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:37.718 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:37.718 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:38.183 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2023-12-06T07:55:38.185 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2023-12-06T07:55:38.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-06T07:55:38.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-06T07:55:38.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-06T07:55:38.376 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:38.377 INFO:tasks.ceph:role: osd.2 2023-12-06T07:55:38.377 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:55:38.377 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:38.451 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:38.452 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:38.452 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:38.452 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:38.452 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:38.931 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2023-12-06T07:55:38.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2023-12-06T07:55:38.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-06T07:55:39.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-06T07:55:39.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-06T07:55:39.172 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:39.172 INFO:tasks.ceph:role: osd.3 2023-12-06T07:55:39.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T07:55:39.173 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-06T07:55:39.247 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:39.248 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:39.743 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2023-12-06T07:55:39.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2023-12-06T07:55:39.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-06T07:55:39.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-06T07:55:39.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:39.993 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.992+0000 7f8114e12500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-06T07:55:39.993 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.993+0000 7f8114e12500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-06T07:55:39.993 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.993+0000 7f8114e12500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T07:55:39.993 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.993+0000 7f8114e12500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T07:55:39.993 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.993+0000 7f8114e12500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-06T07:55:39.994 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:39.993+0000 7f8114e12500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-06T07:55:42.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-06T07:55:42.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.961+0000 7fdf2780e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-06T07:55:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.962+0000 7fdf2780e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-06T07:55:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.962+0000 7fdf2780e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T07:55:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.962+0000 7fdf2780e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T07:55:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.962+0000 7fdf2780e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-06T07:55:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:42.962+0000 7fdf2780e500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-06T07:55:45.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-06T07:55:45.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:45.931 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-06T07:55:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-06T07:55:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T07:55:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T07:55:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-06T07:55:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:45.931+0000 7fad18221500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-06T07:55:48.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-06T07:55:48.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-06T07:55:48.897 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T07:55:48.896+0000 7ff148172500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-06T07:55:51.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-06T07:55:51.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-06T07:55:51.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:51.807 INFO:tasks.ceph:role: osd.4 2023-12-06T07:55:51.808 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:55:51.808 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:51.886 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:51.887 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:51.887 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:51.887 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:52.395 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-06T07:55:52.397 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-06T07:55:52.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-06T07:55:52.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-06T07:55:52.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-06T07:55:52.654 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:52.654 INFO:tasks.ceph:role: osd.5 2023-12-06T07:55:52.654 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:55:52.654 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T07:55:52.730 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:52.731 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:53.173 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-06T07:55:53.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-06T07:55:53.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-06T07:55:53.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-06T07:55:53.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-06T07:55:53.366 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:53.366 INFO:tasks.ceph:role: osd.6 2023-12-06T07:55:53.367 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:55:53.367 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:53.936 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-06T07:55:53.938 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-06T07:55:53.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-06T07:55:54.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-06T07:55:54.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-06T07:55:54.158 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-06T07:55:54.158 INFO:tasks.ceph:role: osd.7 2023-12-06T07:55:54.158 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T07:55:54.158 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-06T07:55:54.233 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T07:55:54.233 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T07:55:54.233 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T07:55:54.233 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-06T07:55:54.233 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T07:55:54.234 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-12-06T07:55:54.234 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T07:55:54.234 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-06T07:55:54.234 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T07:55:54.234 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T07:55:54.747 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-12-06T07:55:54.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-12-06T07:55:54.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-06T07:55:54.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-06T07:55:54.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.037+0000 7faaacc0d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-06T07:55:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.037+0000 7faaacc0d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-06T07:55:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.037+0000 7faaacc0d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T07:55:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.037+0000 7faaacc0d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T07:55:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.037+0000 7faaacc0d500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-06T07:55:55.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:55.038+0000 7faaacc0d500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-06T07:55:57.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-06T07:55:57.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:55:58.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.018+0000 7f9f3138b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-06T07:55:58.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.018+0000 7f9f3138b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-06T07:55:58.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.018+0000 7f9f3138b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T07:55:58.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.018+0000 7f9f3138b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T07:55:58.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.018+0000 7f9f3138b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-06T07:55:58.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:55:58.019+0000 7f9f3138b500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-06T07:56:00.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-06T07:56:00.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.994+0000 7fdf61a0f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.994+0000 7fdf61a0f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.994+0000 7fdf61a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.994+0000 7fdf61a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.994+0000 7fdf61a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-06T07:56:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:00.995+0000 7fdf61a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-06T07:56:03.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-06T07:56:03.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:56:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-06T07:56:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-06T07:56:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T07:56:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T07:56:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-06T07:56:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-12-06T07:56:03.956+0000 7f89d96da500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-06T07:56:06.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-06T07:56:06.827 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-06T07:56:06.828 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:06.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-06T07:56:06.860 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:06.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-06T07:56:06.931 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:06.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-06T07:56:07.000 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:07.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-06T07:56:07.068 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:07.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-06T07:56:07.137 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-06T07:56:07.168 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-06T07:56:07.240 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-06T07:56:07.310 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-06T07:56:07.379 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-06T07:56:07.448 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:07.448 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-06T07:56:07.464 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:07.464 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-06T07:56:07.504 INFO:tasks.ceph:Adding keys to all mons... 2023-12-06T07:56:07.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-06T07:56:07.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-06T07:56:07.547 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.y] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD4KHBl3enhJhAA/Tw+tBc4VoRl/U8fgMDUKA== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD7KHBl0v45OxAAJtbcOzZr9vxg69xIa7dNxA== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD+KHBlDf5iORAACk7jhy3iE17WaF/vMUmgdQ== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQABKXBlYkiMNxAAWK+AfPLGzNBcURsE8Hv0FQ== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAEKXBly/d3NRAABrDhNLYs0Url9EgLLk8cvg== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD4KHBlGVt1KhAAgEEY4PwTAKORyWYA7fRDvg== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQALKXBljl9BAhAAlld9r1u4pl+Lt2cC8TtBeQ== 2023-12-06T07:56:07.548 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAOKXBlwVwfARAAv/xMNeECiaHNfTbS+dG3VA== 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAQKXBlaapIOxAAniquXorokUiprd2HZFeKtg== 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQATKXBlLJ4QORAAE7r0AvNXrpk6WJl8+wAvxg== 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD4KHBlo+YjLRAAGg911dL9Ktd+jZfyNW7yaQ== 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout:[client.1] 2023-12-06T07:56:07.549 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQD4KHBl5KyKMBAArpWP6Z2CTSmizHCsLdHfKQ== 2023-12-06T07:56:07.570 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2023-12-06T07:56:07.570 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD4KHBl3enhJhAA/Tw+tBc4VoRl/U8fgMDUKA== 2023-12-06T07:56:07.570 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2023-12-06T07:56:07.570 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD7KHBl0v45OxAAJtbcOzZr9vxg69xIa7dNxA== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD+KHBlDf5iORAACk7jhy3iE17WaF/vMUmgdQ== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQABKXBlYkiMNxAAWK+AfPLGzNBcURsE8Hv0FQ== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAEKXBly/d3NRAABrDhNLYs0Url9EgLLk8cvg== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD4KHBlGVt1KhAAgEEY4PwTAKORyWYA7fRDvg== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQALKXBljl9BAhAAlld9r1u4pl+Lt2cC8TtBeQ== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAOKXBlwVwfARAAv/xMNeECiaHNfTbS+dG3VA== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAQKXBlaapIOxAAniquXorokUiprd2HZFeKtg== 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2023-12-06T07:56:07.571 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQATKXBlLJ4QORAAE7r0AvNXrpk6WJl8+wAvxg== 2023-12-06T07:56:07.572 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-12-06T07:56:07.572 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD4KHBlo+YjLRAAGg911dL9Ktd+jZfyNW7yaQ== 2023-12-06T07:56:07.572 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2023-12-06T07:56:07.572 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD4KHBl5KyKMBAArpWP6Z2CTSmizHCsLdHfKQ== 2023-12-06T07:56:07.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T07:56:07.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T07:56:07.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T07:56:07.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T07:56:07.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:07.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:08.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:08.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:08.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:08.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T07:56:08.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T07:56:08.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T07:56:08.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T07:56:08.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T07:56:08.260 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-06T07:56:08.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-06T07:56:08.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T07:56:08.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-06T07:56:08.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-06T07:56:08.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T07:56:08.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-06T07:56:08.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-06T07:56:08.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-06T07:56:08.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-06T07:56:08.752 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:56:08.754 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-06T07:56:08.807 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-06T07:56:08.807 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T07:56:08.808 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T07:56:08.808 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T07:56:08.810 INFO:tasks.ceph.mon.a:Started 2023-12-06T07:56:08.810 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T07:56:08.810 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T07:56:08.810 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T07:56:08.813 INFO:tasks.ceph.mon.c:Started 2023-12-06T07:56:08.813 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-06T07:56:08.813 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T07:56:08.813 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T07:56:08.850 INFO:tasks.ceph.mon.b:Started 2023-12-06T07:56:08.850 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-06T07:56:08.850 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-06T07:56:08.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-06T07:56:08.853 INFO:tasks.ceph.mgr.y:Started 2023-12-06T07:56:08.853 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-06T07:56:08.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-06T07:56:08.855 INFO:tasks.ceph.mgr.x:Started 2023-12-06T07:56:08.855 INFO:tasks.ceph:Setting crush tunables to default 2023-12-06T07:56:08.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd crush tunables default 2023-12-06T07:56:16.433 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:07.481 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:16.747 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:07.795 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:16.755 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:07.834 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:16.756 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:07.835 86686== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:17.080 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.159 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:17.081 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.160 86686== Warning: unimplemented fcntl command: 1036 2023-12-06T07:56:33.562 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:33.562+0000 7f08aa00d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-06T07:56:33.613 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:33.612+0000 7f08aa00d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T07:56:33.630 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:33.629+0000 7fd5a602e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-06T07:56:33.684 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:33.684+0000 7fd5a602e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T07:56:33.818 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2023-12-06T07:56:33.831 INFO:tasks.ceph:check_enable_crimson: False 2023-12-06T07:56:33.831 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-06T07:56:33.831 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:33.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-06T07:56:33.861 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:33.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-06T07:56:33.928 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:33.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-06T07:56:33.996 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2023-12-06T07:56:33.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-06T07:56:34.062 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:34.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-06T07:56:34.089 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:34.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-06T07:56:34.145 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.145+0000 7f08aa00d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T07:56:34.156 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:34.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-06T07:56:34.199 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.199+0000 7f08aa00d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T07:56:34.223 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-06T07:56:34.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-06T07:56:34.235 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.235+0000 7fd5a602e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T07:56:34.247 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.246+0000 7f08aa00d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T07:56:34.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 0da60772-f74a-424c-b41a-462ed8a484ac 0 2023-12-06T07:56:34.286 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.286+0000 7fd5a602e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T07:56:34.309 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.308+0000 7f08aa00d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T07:56:34.330 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.330+0000 7fd5a602e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T07:56:34.353 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.352+0000 7f08aa00d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T07:56:34.415 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.415+0000 7fd5a602e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T07:56:34.459 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.459+0000 7fd5a602e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T07:56:34.460 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.460+0000 7f08aa00d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-06T07:56:34.533 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.532+0000 7f08aa00d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T07:56:34.562 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.559+0000 7fd5a602e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-06T07:56:34.580 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.579+0000 7f08aa00d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T07:56:34.633 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.633+0000 7f08aa00d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T07:56:34.636 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.636+0000 7fd5a602e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T07:56:34.679 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:34.679+0000 7f08aa00d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-06T07:56:34.686 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.686+0000 7fd5a602e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T07:56:34.744 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.744+0000 7fd5a602e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T07:56:34.776 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-12-06T07:56:34.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new aa3bcc32-a99b-4382-b6d8-a9561dc53073 1 2023-12-06T07:56:34.794 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:34.794+0000 7fd5a602e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-06T07:56:35.085 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.084+0000 7f08aa00d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T07:56:35.180 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-12-06T07:56:35.189 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.189+0000 7fd5a602e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T07:56:35.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new ea5cc880-085f-4d6a-9307-212ab3aee1ce 2 2023-12-06T07:56:35.225 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.224+0000 7f08aa00d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T07:56:35.326 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.326+0000 7fd5a602e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T07:56:35.357 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.357+0000 7f08aa00d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T07:56:35.404 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.404+0000 7f08aa00d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T07:56:35.449 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.449+0000 7fd5a602e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T07:56:35.494 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.494+0000 7fd5a602e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T07:56:35.504 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.504+0000 7f08aa00d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T07:56:35.557 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-12-06T07:56:35.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new f3f7cd68-c256-4e03-9a53-5a88e9e19c12 3 2023-12-06T07:56:35.603 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.602+0000 7f08aa00d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T07:56:35.604 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.603+0000 7fd5a602e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T07:56:35.697 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.697+0000 7fd5a602e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T07:56:35.751 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.750+0000 7f08aa00d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T07:56:35.795 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.794+0000 7f08aa00d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T07:56:35.849 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.849+0000 7fd5a602e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T07:56:35.894 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:35.894+0000 7fd5a602e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T07:56:35.948 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:35.947+0000 7f08aa00d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T07:56:35.969 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-12-06T07:56:35.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new d078280f-f9dc-490e-a13f-4714b9d70ca8 4 2023-12-06T07:56:36.041 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:36.041+0000 7fd5a602e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T07:56:36.337 INFO:teuthology.orchestra.run.smithi107.stdout:4 2023-12-06T07:56:36.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 03c3b27d-6041-413a-a1d5-807f1349decb 5 2023-12-06T07:56:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:5 2023-12-06T07:56:36.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 332ffee8-2eca-481a-a2a4-ca5edfbebfe2 6 2023-12-06T07:56:36.779 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:36.778+0000 7f08aa00d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T07:56:37.061 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:37.060+0000 7f08aa00d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T07:56:37.088 INFO:teuthology.orchestra.run.smithi107.stdout:6 2023-12-06T07:56:37.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 6eab830f-d790-454c-9c56-5092fab218c4 7 2023-12-06T07:56:37.114 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:37.114+0000 7f08aa00d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T07:56:37.155 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:37.155+0000 7fd5a602e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T07:56:37.222 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:37.222+0000 7f08aa00d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T07:56:37.432 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:37.432+0000 7fd5a602e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T07:56:37.503 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:37.503+0000 7fd5a602e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T07:56:37.614 INFO:tasks.ceph.mgr.y.smithi084.stderr:2023-12-06T07:56:37.614+0000 7fd5a602e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T07:56:37.879 INFO:teuthology.orchestra.run.smithi107.stdout:7 2023-12-06T07:56:37.894 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.894 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-06T07:56:37.894 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-06T07:56:37.896 INFO:tasks.ceph.osd.0:Started 2023-12-06T07:56:37.896 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.897 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-06T07:56:37.897 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-06T07:56:37.899 INFO:tasks.ceph.osd.1:Started 2023-12-06T07:56:37.899 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.899 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-06T07:56:37.900 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-06T07:56:37.901 INFO:tasks.ceph.osd.2:Started 2023-12-06T07:56:37.901 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.902 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-06T07:56:37.902 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-06T07:56:37.904 INFO:tasks.ceph.osd.3:Started 2023-12-06T07:56:37.904 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.904 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-06T07:56:37.904 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-06T07:56:37.906 INFO:tasks.ceph.osd.4:Started 2023-12-06T07:56:37.906 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.906 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-06T07:56:37.906 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-06T07:56:37.908 INFO:tasks.ceph.osd.5:Started 2023-12-06T07:56:37.908 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.909 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-06T07:56:37.909 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-06T07:56:37.910 INFO:tasks.ceph.osd.6:Started 2023-12-06T07:56:37.911 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-06T07:56:37.911 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-06T07:56:37.911 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-06T07:56:37.913 INFO:tasks.ceph.osd.7:Started 2023-12-06T07:56:37.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T07:56:38.371 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:56:38.372 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:56:38.385 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-06T07:56:38.385 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-06T07:56:39.192 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:56:39.382 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:39.381+0000 7f084b0d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-06T07:56:39.492 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T07:56:39.491+0000 7f084b0d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-06T07:56:40.567 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:56:40.567 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:56:40.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T07:56:45.888 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:07.909 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:45.959 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:07.987 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:45.960 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:07.981 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.022 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:08.050 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.226 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:08.237 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.252 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:08.261 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.255 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:08.291 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.285 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:08.296 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.309 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:08.318 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.319 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:08.355 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.332 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:08.358 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.352 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:08.383 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.365 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:08.403 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.391 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:08.417 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.411 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:08.442 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.424 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:08.462 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.487 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:56:46.481+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.571 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:56:46.565+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.855 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:56:46.845+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.971 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:56:46.858+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.972 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:56:46.870+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.972 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:56:46.935+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.972 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:08.911 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:46.972 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:56:46.955+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.977 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:56:46.971+0000 5dc2340 -1 Falling back to public interface 2023-12-06T07:56:46.981 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:09.009 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.251 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:09.272 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.267 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:09.303 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.278 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.289 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.306 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:09.315 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.345 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:09.373 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.356 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.382 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.391 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.422 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.403 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.441 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.473 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:56:47.628 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:09.648 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.633 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:09.669 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.656 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.667 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.680 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:09.689 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.714 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:09.742 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.727 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:09.753 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.757 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:09.788 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.763 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:09.802 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.769 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.780 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.931 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:09.952 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:47.995 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.030 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.013 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:10.041 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.034 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:10.043 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.049 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.076 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.057 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.068 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.067 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:56:48.067 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:56:48.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T07:56:48.085 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.124 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.087 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.119 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.218 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:10.238 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.277 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:10.305 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.283 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.319 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.320 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.347 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.322 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.332 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.341 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:10.350 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.359 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.397 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.360 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.390 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.482 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:10.503 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.544 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:10.572 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.551 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.587 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.586 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.596 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.587 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.614 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.611 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:10.620 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.632 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.670 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.640 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.671 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.746 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:10.766 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.811 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:10.839 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.815 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:10.851 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.854 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.864 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.860 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.883 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.883 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:10.892 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.894 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:10.918 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.905 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:10.944 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:48.915 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:10.947 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.010 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:11.031 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.079 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:11.107 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.090 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.126 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.121 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:11.131 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.157 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:11.166 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.169 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:11.195 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.193 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.231 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.211 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:11.242 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.301 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:11.321 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.369 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:11.397 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.380 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.416 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.415 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:11.426 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.448 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:11.457 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.457 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:11.483 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.484 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.522 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.515 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:11.546 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.668 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:11.704 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.743 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:11.752 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.788 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:11.826 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.819 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:11.839 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.822 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:11.853 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.875 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:11.895 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.889 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:11.917 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.941 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:11.952 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.945 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:11.973 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:49.977 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:12.003 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.020 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:12.031 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.031 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:12.057 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.265 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:12.301 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.320 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:12.355 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.337 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:12.346 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.353 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:12.384 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.368 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:12.406 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.392 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:12.401 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.408 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:12.439 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:50.423 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:12.461 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:54.886 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:56:55.223 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:17.234 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:55.263 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:17.280 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:55.444 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:17.473 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:55.490 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:17.499 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:55.707 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:56:55.708 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:56:55.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T07:56:56.043 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:18.069 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:56.153 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:18.184 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:56.390 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:18.428 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:56.418 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:18.454 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:59.743 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:21.763 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:56:59.747 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:21.767 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.058 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:22.068 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.061 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:22.072 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.101 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:22.129 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.104 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:22.133 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.124 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:22.133 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:00.130 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:22.140 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.098 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:23.129 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.102 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:23.133 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.127 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:23.153 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.131 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:23.158 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.196 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:23.234 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.201 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:23.238 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.405 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:23.438 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:01.406 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:23.442 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:02.532 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:03.339 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:57:03.339 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:03.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T07:57:04.075 INFO:tasks.ceph.osd.7.smithi107.stderr:==00:00:00:26.095 86711== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:04.384 INFO:tasks.ceph.osd.4.smithi107.stderr:==00:00:00:26.412 86710== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:04.580 INFO:tasks.ceph.osd.6.smithi107.stderr:==00:00:00:26.589 86713== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:04.607 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:26.617 86712== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:05.477 INFO:tasks.ceph.osd.3.smithi084.stderr:==00:00:00:27.508 87002== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:05.478 INFO:tasks.ceph.osd.2.smithi084.stderr:==00:00:00:27.504 87003== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:05.521 INFO:tasks.ceph.osd.1.smithi084.stderr:==00:00:00:27.559 87000== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:05.612 INFO:tasks.ceph.osd.0.smithi084.stderr:==00:00:00:27.648 87001== Warning: unimplemented fcntl command: 1038 2023-12-06T07:57:10.163 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:10.856 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:57:10.856 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:56:37.562649+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:10.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-06T07:57:11.674 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:11.671+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-06T07:57:12.427 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:12.424+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-06T07:57:12.431 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:12.421+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-06T07:57:12.624 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:12.574+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-06T07:57:13.288 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:13.285+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-06T07:57:13.298 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:13.295+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-06T07:57:13.466 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:13.463+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-06T07:57:13.653 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:13.650+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-06T07:57:14.489 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:14.481+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2023-12-06T07:57:14.888 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:14.874+0000 12a58640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:15.599 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:15.595+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2023-12-06T07:57:15.670 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:15.666+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2023-12-06T07:57:15.705 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:15.700+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2023-12-06T07:57:16.006 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:15.992+0000 12a58640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:16.053 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:16.040+0000 12257640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:16.104 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:16.086+0000 12257640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:16.583 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:16.580+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2023-12-06T07:57:16.622 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:16.619+0000 c493640 -1 osd.2 0 waiting for initial osdmap 2023-12-06T07:57:16.693 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:16.690+0000 c493640 -1 osd.3 0 waiting for initial osdmap 2023-12-06T07:57:16.747 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:16.743+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2023-12-06T07:57:17.141 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:17.128+0000 12a58640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:17.147 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:17.128+0000 12257640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:17.148 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:17.130+0000 11a56640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:17.236 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:17.222+0000 12257640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T07:57:17.679 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:18.315 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:57:18.315 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":15,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:57:17.264245+0000","last_up_change":"2023-12-06T07:57:17.264245+0000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":933623275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":933623275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":933623275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":933623275}]},"public_addr":"172.21.15.107:6809/933623275","cluster_addr":"172.21.15.107:6810/933623275","heartbeat_back_addr":"172.21.15.107:6812/933623275","heartbeat_front_addr":"172.21.15.107:6811/933623275","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:18.326 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-12-06T07:57:25.133 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:25.791 INFO:teuthology.misc.health.smithi084.stdout: 2023-12-06T07:57:25.791 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":18,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:57:21.825411+0000","last_up_change":"2023-12-06T07:57:18.326272+0000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":933623275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":933623275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":933623275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":933623275}]},"public_addr":"172.21.15.107:6809/933623275","cluster_addr":"172.21.15.107:6810/933623275","heartbeat_back_addr":"172.21.15.107:6812/933623275","heartbeat_front_addr":"172.21.15.107:6811/933623275","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:25.803 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-06T07:57:25.803 INFO:tasks.ceph:Creating RBD pool 2023-12-06T07:57:25.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-06T07:57:27.961 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'rbd' created 2023-12-06T07:57:27.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-06T07:57:28.802 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:19.881 86686== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:28.877 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:19.925 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:28.888 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:19.967 86686== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:28.893 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:19.972 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:28.932 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:19.980 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:28.964 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:20.044 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:29.162 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:20.241 86686== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:29.163 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:20.210 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:29.273 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:20.352 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:57:29.572 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-06T07:57:29.587 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-06T07:57:29.587 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-06T07:57:29.587 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-06T07:57:29.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-06T07:57:29.588 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-06T07:57:30.389 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:30.401 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2063299331}]},"active_addr":"172.21.15.107:6800/2063299331","active_change":"2023-12-06T07:56:37.518190+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3459463790}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2639546867}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2060349495}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3910365677}]}]} 2023-12-06T07:57:30.405 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-06T07:57:30.405 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-06T07:57:30.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:31.167 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:31.167 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:57:30.451523+0000","last_up_change":"2023-12-06T07:57:18.326272+0000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":933623275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":933623275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":933623275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":933623275}]},"public_addr":"172.21.15.107:6809/933623275","cluster_addr":"172.21.15.107:6810/933623275","heartbeat_back_addr":"172.21.15.107:6812/933623275","heartbeat_front_addr":"172.21.15.107:6811/933623275","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:31.182 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-06T07:57:31.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:31.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:31.758 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:57:30.451523+0000","last_up_change":"2023-12-06T07:57:18.326272+0000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":933623275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":933623275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":933623275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":933623275}]},"public_addr":"172.21.15.107:6809/933623275","cluster_addr":"172.21.15.107:6810/933623275","heartbeat_back_addr":"172.21.15.107:6812/933623275","heartbeat_front_addr":"172.21.15.107:6811/933623275","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:31.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-06T07:57:31.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-06T07:57:31.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-06T07:57:31.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-06T07:57:31.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-06T07:57:31.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-06T07:57:31.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-06T07:57:31.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-06T07:57:32.154 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:32.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T07:57:32.183 INFO:teuthology.orchestra.run.smithi084.stdout:60129542149 2023-12-06T07:57:32.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T07:57:32.195 INFO:teuthology.orchestra.run.smithi084.stdout:55834574853 2023-12-06T07:57:32.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T07:57:32.210 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:32.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T07:57:32.230 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:32.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T07:57:32.269 INFO:teuthology.orchestra.run.smithi084.stdout:68719476743 2023-12-06T07:57:32.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T07:57:32.382 INFO:teuthology.orchestra.run.smithi084.stdout:60129542150 2023-12-06T07:57:32.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T07:57:32.393 INFO:teuthology.orchestra.run.smithi084.stdout:64424509446 2023-12-06T07:57:32.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T07:57:33.011 INFO:teuthology.orchestra.run.smithi084.stdout:55834574852 2023-12-06T07:57:33.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2023-12-06T07:57:33.079 INFO:teuthology.orchestra.run.smithi084.stdout:60129542148 2023-12-06T07:57:33.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2023-12-06T07:57:33.158 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:33.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.1 2023-12-06T07:57:33.294 INFO:teuthology.orchestra.run.smithi084.stdout:68719476740 2023-12-06T07:57:33.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.3 2023-12-06T07:57:33.374 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:33.387 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-06T07:57:33.403 INFO:teuthology.orchestra.run.smithi084.stdout:64424509445 2023-12-06T07:57:33.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2023-12-06T07:57:33.497 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:33.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2023-12-06T07:57:33.627 INFO:teuthology.orchestra.run.smithi084.stdout:60129542149 2023-12-06T07:57:33.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2023-12-06T07:57:34.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T07:57:34.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T07:57:34.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T07:57:34.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T07:57:34.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T07:57:34.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T07:57:34.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T07:57:34.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T07:57:34.723 INFO:teuthology.orchestra.run.smithi084.stdout:55834574852 2023-12-06T07:57:34.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2023-12-06T07:57:34.933 INFO:teuthology.orchestra.run.smithi084.stdout:60129542148 2023-12-06T07:57:34.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2023-12-06T07:57:35.008 INFO:teuthology.orchestra.run.smithi084.stdout:68719476740 2023-12-06T07:57:35.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.3 2023-12-06T07:57:35.110 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:35.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.1 2023-12-06T07:57:35.189 INFO:teuthology.orchestra.run.smithi084.stdout:64424509445 2023-12-06T07:57:35.193 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:35.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2023-12-06T07:57:35.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2023-12-06T07:57:35.279 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:35.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-06T07:57:35.454 INFO:teuthology.orchestra.run.smithi084.stdout:60129542150 2023-12-06T07:57:35.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.5 2023-12-06T07:57:35.469 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:35.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T07:57:35.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T07:57:36.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T07:57:36.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T07:57:36.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T07:57:36.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T07:57:36.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T07:57:36.515 INFO:teuthology.orchestra.run.smithi084.stdout:55834574853 2023-12-06T07:57:36.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2023-12-06T07:57:36.530 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:60129542149 2023-12-06T07:57:36.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2023-12-06T07:57:36.730 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:36.914 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:36.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.1 2023-12-06T07:57:36.927 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:37.086 INFO:teuthology.orchestra.run.smithi084.stdout:68719476742 2023-12-06T07:57:37.095 INFO:teuthology.orchestra.run.smithi084.stdout:68719476741 2023-12-06T07:57:37.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.2 2023-12-06T07:57:37.101 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:37.113 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.3 2023-12-06T07:57:37.113 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:37.296 INFO:teuthology.orchestra.run.smithi084.stdout:68719476744 2023-12-06T07:57:37.306 INFO:teuthology.orchestra.run.smithi084.stdout:64424509447 2023-12-06T07:57:37.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476744 for osd.0 2023-12-06T07:57:37.310 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:37.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.4 2023-12-06T07:57:37.323 DEBUG:teuthology.parallel:result is None 2023-12-06T07:57:37.323 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T07:57:37.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:57:37.920 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:37.920 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:57:37.932 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-12-06T07:57:37.802901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218420,"kb_used_data":2256,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756300,"statfs":{"total":773094113280,"available":772870451200,"internally_reserved":0,"allocated":2310144,"data_stored":1306448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.283463+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:57:33.283463+0000","last_peered":"2023-12-06T07:57:33.283463+0000","last_clean":"2023-12-06T07:57:33.283463+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:57:33.283463+0000","last_undegraded":"2023-12-06T07:57:33.283463+0000","last_fullsized":"2023-12-06T07:57:33.283463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.261653+0000","last_change":"2023-12-06T07:57:30.401832+0000","last_active":"2023-12-06T07:57:33.261653+0000","last_peered":"2023-12-06T07:57:33.261653+0000","last_clean":"2023-12-06T07:57:33.261653+0000","last_became_active":"2023-12-06T07:57:30.374582+0000","last_became_peered":"2023-12-06T07:57:30.374582+0000","last_unstale":"2023-12-06T07:57:33.261653+0000","last_undegraded":"2023-12-06T07:57:33.261653+0000","last_fullsized":"2023-12-06T07:57:33.261653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:20.205766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.287028+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:57:33.287028+0000","last_peered":"2023-12-06T07:57:33.287028+0000","last_clean":"2023-12-06T07:57:33.287028+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:57:33.287028+0000","last_undegraded":"2023-12-06T07:57:33.287028+0000","last_fullsized":"2023-12-06T07:57:33.287028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.263320+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:57:33.263320+0000","last_peered":"2023-12-06T07:57:33.263320+0000","last_clean":"2023-12-06T07:57:33.263320+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:57:33.263320+0000","last_undegraded":"2023-12-06T07:57:33.263320+0000","last_fullsized":"2023-12-06T07:57:33.263320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.273444+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:57:33.273444+0000","last_peered":"2023-12-06T07:57:33.273444+0000","last_clean":"2023-12-06T07:57:33.273444+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:57:33.273444+0000","last_undegraded":"2023-12-06T07:57:33.273444+0000","last_fullsized":"2023-12-06T07:57:33.273444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.223429+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:57:32.223429+0000","last_peered":"2023-12-06T07:57:32.223429+0000","last_clean":"2023-12-06T07:57:32.223429+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:57:32.223429+0000","last_undegraded":"2023-12-06T07:57:32.223429+0000","last_fullsized":"2023-12-06T07:57:32.223429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.262252+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:57:33.262252+0000","last_peered":"2023-12-06T07:57:33.262252+0000","last_clean":"2023-12-06T07:57:33.262252+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:57:33.262252+0000","last_undegraded":"2023-12-06T07:57:33.262252+0000","last_fullsized":"2023-12-06T07:57:33.262252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.277535+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:57:33.277535+0000","last_peered":"2023-12-06T07:57:33.277535+0000","last_clean":"2023-12-06T07:57:33.277535+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:57:33.277535+0000","last_undegraded":"2023-12-06T07:57:33.277535+0000","last_fullsized":"2023-12-06T07:57:33.277535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T07:57:30.603038+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:57:30.603038+0000","last_peered":"2023-12-06T07:57:30.603038+0000","last_clean":"2023-12-06T07:57:30.603038+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:57:30.603038+0000","last_undegraded":"2023-12-06T07:57:30.603038+0000","last_fullsized":"2023-12-06T07:57:30.603038+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.042}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5510000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.988}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.93}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.537000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.741}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.056999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1140000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.494}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.722000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.967000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.372999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.268999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.866}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.881}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3030000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.686999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.239999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.503}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7170000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.518000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.209}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.895}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.434999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.93}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4089999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7720000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.667999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4529999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.385}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.383}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.342000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.911}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.353999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.178000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9729999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.827999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.657}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.521000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.407}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.242999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8940000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.493}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.333}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:57:37.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:57:38.513 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:38.513 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:57:38.525 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-12-06T07:57:37.802901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218420,"kb_used_data":2256,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756300,"statfs":{"total":773094113280,"available":772870451200,"internally_reserved":0,"allocated":2310144,"data_stored":1306448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.283463+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:57:33.283463+0000","last_peered":"2023-12-06T07:57:33.283463+0000","last_clean":"2023-12-06T07:57:33.283463+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:57:33.283463+0000","last_undegraded":"2023-12-06T07:57:33.283463+0000","last_fullsized":"2023-12-06T07:57:33.283463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.261653+0000","last_change":"2023-12-06T07:57:30.401832+0000","last_active":"2023-12-06T07:57:33.261653+0000","last_peered":"2023-12-06T07:57:33.261653+0000","last_clean":"2023-12-06T07:57:33.261653+0000","last_became_active":"2023-12-06T07:57:30.374582+0000","last_became_peered":"2023-12-06T07:57:30.374582+0000","last_unstale":"2023-12-06T07:57:33.261653+0000","last_undegraded":"2023-12-06T07:57:33.261653+0000","last_fullsized":"2023-12-06T07:57:33.261653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:20.205766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.287028+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:57:33.287028+0000","last_peered":"2023-12-06T07:57:33.287028+0000","last_clean":"2023-12-06T07:57:33.287028+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:57:33.287028+0000","last_undegraded":"2023-12-06T07:57:33.287028+0000","last_fullsized":"2023-12-06T07:57:33.287028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.263320+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:57:33.263320+0000","last_peered":"2023-12-06T07:57:33.263320+0000","last_clean":"2023-12-06T07:57:33.263320+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:57:33.263320+0000","last_undegraded":"2023-12-06T07:57:33.263320+0000","last_fullsized":"2023-12-06T07:57:33.263320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.273444+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:57:33.273444+0000","last_peered":"2023-12-06T07:57:33.273444+0000","last_clean":"2023-12-06T07:57:33.273444+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:57:33.273444+0000","last_undegraded":"2023-12-06T07:57:33.273444+0000","last_fullsized":"2023-12-06T07:57:33.273444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.223429+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:57:32.223429+0000","last_peered":"2023-12-06T07:57:32.223429+0000","last_clean":"2023-12-06T07:57:32.223429+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:57:32.223429+0000","last_undegraded":"2023-12-06T07:57:32.223429+0000","last_fullsized":"2023-12-06T07:57:32.223429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.262252+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:57:33.262252+0000","last_peered":"2023-12-06T07:57:33.262252+0000","last_clean":"2023-12-06T07:57:33.262252+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:57:33.262252+0000","last_undegraded":"2023-12-06T07:57:33.262252+0000","last_fullsized":"2023-12-06T07:57:33.262252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.277535+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:57:33.277535+0000","last_peered":"2023-12-06T07:57:33.277535+0000","last_clean":"2023-12-06T07:57:33.277535+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:57:33.277535+0000","last_undegraded":"2023-12-06T07:57:33.277535+0000","last_fullsized":"2023-12-06T07:57:33.277535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T07:57:30.603038+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:57:30.603038+0000","last_peered":"2023-12-06T07:57:30.603038+0000","last_clean":"2023-12-06T07:57:30.603038+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:57:30.603038+0000","last_undegraded":"2023-12-06T07:57:30.603038+0000","last_fullsized":"2023-12-06T07:57:30.603038+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.042}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5510000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.988}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.93}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.537000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.741}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.056999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1140000000000008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.494}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.722000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.967000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.372999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.268999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.866}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.881}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3030000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.686999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.239999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.503}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7170000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.518000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.209}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.895}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.434999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.93}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4089999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7720000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.667999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4529999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.385}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.383}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.342000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.911}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.353999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.178000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9729999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.827999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.657}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.521000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.407}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.242999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8940000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.493}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.333}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:57:38.526 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T07:57:38.526 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-06T07:57:38.527 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-06T07:57:38.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-06T07:57:39.125 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:39.126 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-06T07:57:39.139 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-06T07:57:39.140 INFO:teuthology.run_tasks:Running task thrashosds... 2023-12-06T07:57:39.151 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-12-06T07:57:39.151 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sseshasa-testing-2023-12-04-1848'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sseshasa-testing-2023-12-04-1848', 'sha1': '96fa2578c622ca9c169af6f426ca8b1616130e58'}} 2023-12-06T07:57:39.151 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-12-06T07:57:39.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-12-06T07:57:39.930 INFO:tasks.thrashosds:Beginning thrashosds... 2023-12-06T07:57:39.930 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T07:57:39.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:57:40.489 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:40.490 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:57:40.501 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-12-06T07:57:39.803278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218420,"kb_used_data":2256,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756300,"statfs":{"total":773094113280,"available":772870451200,"internally_reserved":0,"allocated":2310144,"data_stored":1306448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.323604"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.283463+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:57:33.283463+0000","last_peered":"2023-12-06T07:57:33.283463+0000","last_clean":"2023-12-06T07:57:33.283463+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:57:33.283463+0000","last_undegraded":"2023-12-06T07:57:33.283463+0000","last_fullsized":"2023-12-06T07:57:33.283463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.261653+0000","last_change":"2023-12-06T07:57:30.401832+0000","last_active":"2023-12-06T07:57:33.261653+0000","last_peered":"2023-12-06T07:57:33.261653+0000","last_clean":"2023-12-06T07:57:33.261653+0000","last_became_active":"2023-12-06T07:57:30.374582+0000","last_became_peered":"2023-12-06T07:57:30.374582+0000","last_unstale":"2023-12-06T07:57:33.261653+0000","last_undegraded":"2023-12-06T07:57:33.261653+0000","last_fullsized":"2023-12-06T07:57:33.261653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:20.205766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.287028+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:57:33.287028+0000","last_peered":"2023-12-06T07:57:33.287028+0000","last_clean":"2023-12-06T07:57:33.287028+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:57:33.287028+0000","last_undegraded":"2023-12-06T07:57:33.287028+0000","last_fullsized":"2023-12-06T07:57:33.287028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.263320+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:57:33.263320+0000","last_peered":"2023-12-06T07:57:33.263320+0000","last_clean":"2023-12-06T07:57:33.263320+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:57:33.263320+0000","last_undegraded":"2023-12-06T07:57:33.263320+0000","last_fullsized":"2023-12-06T07:57:33.263320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.273444+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:57:33.273444+0000","last_peered":"2023-12-06T07:57:33.273444+0000","last_clean":"2023-12-06T07:57:33.273444+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:57:33.273444+0000","last_undegraded":"2023-12-06T07:57:33.273444+0000","last_fullsized":"2023-12-06T07:57:33.273444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.223429+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:57:32.223429+0000","last_peered":"2023-12-06T07:57:32.223429+0000","last_clean":"2023-12-06T07:57:32.223429+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:57:32.223429+0000","last_undegraded":"2023-12-06T07:57:32.223429+0000","last_fullsized":"2023-12-06T07:57:32.223429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.262252+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:57:33.262252+0000","last_peered":"2023-12-06T07:57:33.262252+0000","last_clean":"2023-12-06T07:57:33.262252+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:57:33.262252+0000","last_undegraded":"2023-12-06T07:57:33.262252+0000","last_fullsized":"2023-12-06T07:57:33.262252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.277535+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:57:33.277535+0000","last_peered":"2023-12-06T07:57:33.277535+0000","last_clean":"2023-12-06T07:57:33.277535+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:57:33.277535+0000","last_undegraded":"2023-12-06T07:57:33.277535+0000","last_fullsized":"2023-12-06T07:57:33.277535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T07:57:30.603038+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:57:30.603038+0000","last_peered":"2023-12-06T07:57:30.603038+0000","last_clean":"2023-12-06T07:57:30.603038+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:57:30.603038+0000","last_undegraded":"2023-12-06T07:57:30.603038+0000","last_fullsized":"2023-12-06T07:57:30.603038+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.042}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5510000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.988}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.93}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.537000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.741}]}]},{"osd":2,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.776}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9909999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8330000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.469000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.624000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.672000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.277999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7519999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.143000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.715999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.9}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.808999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.138000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2709999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.243}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9860000000000007}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2639999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.536999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.308}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.273}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4089999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7720000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.667999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4529999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.385}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.383}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.342000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.911}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.353999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.178000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9729999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.827999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.657}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.521000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.407}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.242999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8940000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.493}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.333}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:57:40.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:57:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:41.070 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:57:41.083 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-12-06T07:57:39.803278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218420,"kb_used_data":2256,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756300,"statfs":{"total":773094113280,"available":772870451200,"internally_reserved":0,"allocated":2310144,"data_stored":1306448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.323604"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.283463+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:57:33.283463+0000","last_peered":"2023-12-06T07:57:33.283463+0000","last_clean":"2023-12-06T07:57:33.283463+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:57:33.283463+0000","last_undegraded":"2023-12-06T07:57:33.283463+0000","last_fullsized":"2023-12-06T07:57:33.283463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.261653+0000","last_change":"2023-12-06T07:57:30.401832+0000","last_active":"2023-12-06T07:57:33.261653+0000","last_peered":"2023-12-06T07:57:33.261653+0000","last_clean":"2023-12-06T07:57:33.261653+0000","last_became_active":"2023-12-06T07:57:30.374582+0000","last_became_peered":"2023-12-06T07:57:30.374582+0000","last_unstale":"2023-12-06T07:57:33.261653+0000","last_undegraded":"2023-12-06T07:57:33.261653+0000","last_fullsized":"2023-12-06T07:57:33.261653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:20.205766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.287028+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:57:33.287028+0000","last_peered":"2023-12-06T07:57:33.287028+0000","last_clean":"2023-12-06T07:57:33.287028+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:57:33.287028+0000","last_undegraded":"2023-12-06T07:57:33.287028+0000","last_fullsized":"2023-12-06T07:57:33.287028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.263320+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:57:33.263320+0000","last_peered":"2023-12-06T07:57:33.263320+0000","last_clean":"2023-12-06T07:57:33.263320+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:57:33.263320+0000","last_undegraded":"2023-12-06T07:57:33.263320+0000","last_fullsized":"2023-12-06T07:57:33.263320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.273444+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:57:33.273444+0000","last_peered":"2023-12-06T07:57:33.273444+0000","last_clean":"2023-12-06T07:57:33.273444+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:57:33.273444+0000","last_undegraded":"2023-12-06T07:57:33.273444+0000","last_fullsized":"2023-12-06T07:57:33.273444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.223429+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:57:32.223429+0000","last_peered":"2023-12-06T07:57:32.223429+0000","last_clean":"2023-12-06T07:57:32.223429+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:57:32.223429+0000","last_undegraded":"2023-12-06T07:57:32.223429+0000","last_fullsized":"2023-12-06T07:57:32.223429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.262252+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:57:33.262252+0000","last_peered":"2023-12-06T07:57:33.262252+0000","last_clean":"2023-12-06T07:57:33.262252+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:57:33.262252+0000","last_undegraded":"2023-12-06T07:57:33.262252+0000","last_fullsized":"2023-12-06T07:57:33.262252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.277535+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:57:33.277535+0000","last_peered":"2023-12-06T07:57:33.277535+0000","last_clean":"2023-12-06T07:57:33.277535+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:57:33.277535+0000","last_undegraded":"2023-12-06T07:57:33.277535+0000","last_fullsized":"2023-12-06T07:57:33.277535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":118,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-06T07:57:30.603038+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:57:30.603038+0000","last_peered":"2023-12-06T07:57:30.603038+0000","last_clean":"2023-12-06T07:57:30.603038+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:57:30.603038+0000","last_undegraded":"2023-12-06T07:57:30.603038+0000","last_fullsized":"2023-12-06T07:57:30.603038+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.042}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5510000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.988}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.93}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.537000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.741}]}]},{"osd":2,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.776}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9909999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8330000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.469000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.624000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.672000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.277999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7519999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.143000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.715999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.9}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.808999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.138000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2709999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.243}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9860000000000007}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2639999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.536999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.308}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.273}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4089999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7720000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.667999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4529999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.385}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4619999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.383}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.342000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.911}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.353999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.178000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9729999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.827999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.657}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.521000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.407}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.242999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8940000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.493}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.333}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:57:41.084 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T07:57:41.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-12-06T07:57:41.717 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 22 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:fsid aba07bce-2203-46e5-bd3f-65b300a2f9db 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:created 2023-12-06T07:56:32.091884+0000 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2023-12-06T07:57:31.995910+0000 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:crush_version 6 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:full_ratio 0.95 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:backfillfull_ratio 0.9 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:nearfull_ratio 0.85 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:require_min_compat_client luminous 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:min_compat_client jewel 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:require_osd_release reef 2023-12-06T07:57:41.718 INFO:teuthology.orchestra.run.smithi084.stdout:stretch_mode_enabled false 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 application rbd read_balance_score 4.98 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:max_osd 8 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6800/1405545511 v2:172.21.15.84:6801/1405545511 exists,up 0da60772-f74a-424c-b41a-462ed8a484ac 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6812/1973529083 v2:172.21.15.84:6813/1973529083 exists,up aa3bcc32-a99b-4382-b6d8-a9561dc53073 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6804/1651824594 v2:172.21.15.84:6805/1651824594 exists,up ea5cc880-085f-4d6a-9307-212ab3aee1ce 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 up in weight 1 up_from 16 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6808/3951389031 v2:172.21.15.84:6809/3951389031 exists,up f3f7cd68-c256-4e03-9a53-5a88e9e19c12 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6805/2922507264 v2:172.21.15.107:6806/2922507264 exists,up d078280f-f9dc-490e-a13f-4714b9d70ca8 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6809/933623275 v2:172.21.15.107:6810/933623275 exists,up 03c3b27d-6041-413a-a1d5-807f1349decb 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6813/584359350 v2:172.21.15.107:6814/584359350 exists,up 332ffee8-2eca-481a-a2a4-ca5edfbebfe2 2023-12-06T07:57:41.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6801/1873215283 v2:172.21.15.107:6802/1873215283 exists,up 6eab830f-d790-454c-9c56-5092fab218c4 2023-12-06T07:57:41.733 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6800/1405545511 v2:172.21.15.84:6801/1405545511 exists,up 0da60772-f74a-424c-b41a-462ed8a484ac', 'osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6812/1973529083 v2:172.21.15.84:6813/1973529083 exists,up aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6804/1651824594 v2:172.21.15.84:6805/1651824594 exists,up ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'osd.3 up in weight 1 up_from 16 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6808/3951389031 v2:172.21.15.84:6809/3951389031 exists,up f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'osd.4 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6805/2922507264 v2:172.21.15.107:6806/2922507264 exists,up d078280f-f9dc-490e-a13f-4714b9d70ca8', 'osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6809/933623275 v2:172.21.15.107:6810/933623275 exists,up 03c3b27d-6041-413a-a1d5-807f1349decb', 'osd.6 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6813/584359350 v2:172.21.15.107:6814/584359350 exists,up 332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'osd.7 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6801/1873215283 v2:172.21.15.107:6802/1873215283 exists,up 6eab830f-d790-454c-9c56-5092fab218c4'] 2023-12-06T07:57:41.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-12-06T07:57:41.912 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "mon.a", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster": "ceph", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket_mode": "", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_client_required": "cephx, none", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_cluster_required": "cephx", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_debug": "true", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_required": "cephx", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_ticket_ttl": "240.000000", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_supported": "cephx", 2023-12-06T07:57:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio": "true", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_max_queue_depth": "1024", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_poll_ms": "250", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_reap_max": "16", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_async_discard": "false", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_block_size": "4096", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio": "false", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_inflight_ios": "false", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_enable_discard": "false", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry": "3", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry_interval": "0.100000", 2023-12-06T07:57:41.914 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash": "0", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash_flush_delay": "2", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_hipri": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_sqthread_poll": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_buffer_alignment": "4096", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_type": "", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_alloc_size": "1048576", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_allocator": "hybrid", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_buffered_io": "true", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_for_zeros": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_often": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_compact_log_sync": "false", 2023-12-06T07:57:41.915 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_size": "16777216", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_replay_check_allocations": "true", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_log_runway": "4194304", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_prefetch": "1048576", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_flush_size": "524288", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_log_runway": "1048576", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery": "false", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_shared_alloc_size": "65536", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_sync_write": "false", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-12-06T07:57:41.916 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocation_from_file": "true", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocator": "hybrid", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_blobid_prealloc": "10240", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_create": "true", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_create": "false", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_path": "", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_size": "0", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_path": "", 2023-12-06T07:57:41.917 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_preallocate_file": "false", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_size": "107374182400", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_create": "false", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_path": "", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_size": "100663296", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs": "true", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_env_mirror": "false", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T07:57:41.918 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune": "true", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size": "0", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_type": "2q", 2023-12-06T07:57:41.920 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_clone_cow": "true", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_algorithm": "snappy", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size": "0", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size": "0", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_mode": "none", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_csum_type": "crc32c", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_settings": "default", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_freelist": "false", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_fsck_abort": "false", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-12-06T07:57:41.921 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_read_err": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_legacy_omap": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefill": "0.000000", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_small_allocations": "0", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_read": "true", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_write": "false", 2023-12-06T07:57:41.922 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops": "0", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_elastic_shared_blobs": "true", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fail_eio": "false", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs": "true", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-12-06T07:57:41.923 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_ignore_data_csum": "false", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kvbackend": "rocksdb", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-12-06T07:57:41.924 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_op_age": "5.000000", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_alloc_size": "0", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size": "0", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_defer_interval": "3.000000", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_deferred_txc": "32", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size": "0", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_nid_prealloc": "1024", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size": "0", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_qfsck_on_mount": "true", 2023-12-06T07:57:41.925 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_retry_disk_reads": "3", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cf": "true", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options_annex": "", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_coremask": "0x1", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_io_sleep": "5", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_max_io_completion": "0", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_mem": "512", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_sync_submit_transaction": "false", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_bytes": "67108864", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io": "0", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-12-06T07:57:41.926 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_tracing": "false", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved": "0", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_zero_block_detection": "false", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_action_update_interval": "2", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-12-06T07:57:41.927 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_mount_timeout": "10", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_signatures": "false", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_version": "2", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_signatures": "false", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_version": "2", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_signatures": "false", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_version": "2", 2023-12-06T07:57:41.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_sign_messages": "true", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "chdir": "", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_acl_type": "", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_asio_thread_count": "2", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_mid": "0.750000", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_size": "16384", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_caps_release_delay": "5", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_check_pool_perm": "true", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_collect_and_send_global_metrics": "false", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_force_sync_read": "false", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_getattr_caps": "false", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_tick_delay": "0", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_remount": "false", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_dirsize_rbytes": "true", 2023-12-06T07:57:41.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_force_lazyio": "false", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_fs": "", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_fixed_oldest_tid": "false", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_release_failure": "false", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_inline_size": "4096", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_retries_on_remount_failure": "5", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mds_namespace": "", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_metadata": "", 2023-12-06T07:57:41.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_gid": "-1", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_timeout": "300", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_uid": "-1", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mountpoint": "/", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_notify_timeout": "10", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc": "true", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty": "104857600", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty_age": "5.000000", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_objects": "1000", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_size": "209715200", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_target_dirty": "8388608", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_permissions": "true", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota": "true", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota_df": "true", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_bytes": "0", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_periods": "4", 2023-12-06T07:57:41.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_min": "131072", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_reconnect_stale": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_shutdown_timeout": "30", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_snapdir": ".snap", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_tick_interval": "1", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_trace": "", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_try_dentry_invalidate": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_faked_inos": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_random_mds": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_port": "12201", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_monitors": "default=true", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog": "false", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_level": "info", 2023-12-06T07:57:41.932 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "-", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network": "", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network_interface": "", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_isal": "false", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_level": "5", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_winsize": "-15", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zstd_level": "1", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_op_num_threads": "6", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_thread_cpu_cores": "", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_obc_lru_size": "10", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_smp": "1", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "", 2023-12-06T07:57:41.933 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook": "", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook_timeout": "10", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "daemonize": "false", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "daos_pool": "tank", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_name_prefix": "dbstore", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_alienstore": "0/5", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_allow_any_pool_priority": "false", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok": "1/5", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok_assert_abort": "false", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asserts_on_shutdown": "true", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_auth": "20/20", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bdev": "1/3", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluefs": "1/5", 2023-12-06T07:57:41.934 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluestore": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_buffer": "0/1", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_exporter": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephfs_mirror": "0/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephsqlite": "0/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_client": "0/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_compressor": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_context": "0/1", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crimson_interrupt": "0/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crush": "1/1", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crypto": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cyanstore": "0/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_deliberately_leak_memory": "false", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_disable_randomized_ping": "false", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_dpdk": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_eventtrace": "1/5", 2023-12-06T07:57:41.935 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filer": "0/1", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filestore": "1/3", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_finisher": "1/1", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_fuse": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeat_testing_span": "0", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeatmap": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_immutable_obj_cache": "0/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_javaclient": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journal": "1/3", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journaler": "0/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_kstore": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_lockdep": "0/1", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mclock": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_balancer": "1/5", 2023-12-06T07:57:41.936 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_locker": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log_expire": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_migrator": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_memstore": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgr": "2/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgrc": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mon": "20/20", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_monc": "0/10", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ms": "1/1", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_none": "0/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objclass": "0/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objectcacher": "0/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objecter": "0/1", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_optracker": "0/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_osd": "1/5", 2023-12-06T07:57:41.937 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_paxos": "20/20", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_perfcounter": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_prioritycache": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rados": "0/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd": "0/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_mirror": "0/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_pwl": "0/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_replay": "0/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_refs": "0/0", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_reserver": "1/1", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_access": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_datacache": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_dbstore": "1/5", 2023-12-06T07:57:41.938 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_flight": "1/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_sync": "1/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rocksdb": "4/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_backref": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cache": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cleaner": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_device": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_epm": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_journal": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_lba": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_odata": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_omap": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_onode": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_t": "0/5", 2023-12-06T07:57:41.939 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_tm": "0/5", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_striper": "0/1", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_test": "0/5", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_throttle": "1/1", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_timer": "0/1", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_tp": "0/5", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_trace": "1/5", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-12-06T07:57:41.940 INFO:teuthology.orchestra.run.smithi084.stdout: "device_failure_prediction_mode": "none", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_graylog": "false", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_journald": "false", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_stderr": "true", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_syslog": "false", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event_tracing": "false", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_addr": "0.0.0.0", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_http_port": "9926", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_prio_limit": "5", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sort_metrics": "true", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_stats_period": "5", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "fake_statfs_for_testing": "0", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "fatal_signal_handlers": "true", 2023-12-06T07:57:41.941 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_purge_ops": "10", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_truncate_ops": "128", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_apply_finisher_threads": "1", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_blackhole": "false", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_clone_range": "true", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_snap": "true", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_caller_concurrency": "10", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_collect_device_partition_information": "true", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_commit_timeout": "600.000000", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_inject_read_err": "false", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_omap_check": "false", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_random_read_err": "0.000000", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_verify_split": "false", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_dump_file": "", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-12-06T07:57:41.942 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fadvise": "true", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fail_eio": "true", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_shards": "16", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_size": "128", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap": "false", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap_threshold": "4096", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_index_retry_probability": "0.000000", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_inject_stall": "0", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_parallel": "false", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_trailing": "false", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_writeahead": "false", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_kill_at": "0", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size": "0", 2023-12-06T07:57:41.943 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs": "0", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_other": "2", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_sync_interval": "5.000000", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size": "0", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_merge_threshold": "-10", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_min_sync_interval": "0.010000", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_odsync_write": "false", 2023-12-06T07:57:41.944 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend": "rocksdb", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend_path": "", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_header_cache_size": "1024", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_ondisk_finisher_threads": "1", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_timeout": "60", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_threads": "2", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_punch_hole": "false", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_bytes": "104857600", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-12-06T07:57:41.945 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_ops": "50", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_seek_data_hole": "false", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc": "false", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_splice": "false", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_multiple": "2", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_rand_factor": "20", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_update_to": "1000", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-12-06T07:57:41.946 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_enable": "true", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_xfs_extsize": "false", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_zfs_snap": "false", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fio_dir": "/tmp/fio", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_allow_other": "true", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_atomic_o_trunc": "true", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_big_writes": "true", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_debug": "false", 2023-12-06T07:57:41.947 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_default_permissions": "false", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_disable_pagecache": "false", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_max_write": "0", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_multithreaded": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_require_active_mds": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_set_user_groups": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_move": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_read": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_write": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_syncfs_on_mksnap": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_use_invalidate_cb": "true", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_target_name": "ceph", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_file": "", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_inject_failure": "0", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_interval": "5", 2023-12-06T07:57:41.948 INFO:teuthology.orchestra.run.smithi084.stdout: "host": "smithi084", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_size": "1073741824", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_watermark": "0.900000", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "inject_early_sigterm": "false", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_agent_port": "6799", 2023-12-06T07:57:41.949 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_tracing_enable": "false", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_aio": "true", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_align_min_size": "65536", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_align": "true", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_size": "4096", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_dio": "true", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_discard": "false", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_force_aio": "false", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_ignore_corruption": "false", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_bytes": "10485760", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_entries": "100", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_replay_from": "0", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_multiple": "0.000000", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-12-06T07:57:41.950 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_max_multiple": "0.000000", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_write_header_frequency": "0", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_zero_on_create": "false", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prefetch_periods": "10", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prezero_periods": "5", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_write_head_interval": "15", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "key": "", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "keyfile": "", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-12-06T07:57:41.951 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_backend": "rocksdb", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_default_stripe_size": "65536", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount": "false", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount_deep": "true", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_bytes": "67108864", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_ops": "512", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_nid_prealloc": "1024", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_onode_map_size": "1024", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_submit_transaction": "false", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_transaction": "false", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "librados_thread_count": "2", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep": "false", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep_force_backtrace": "false", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "log_coarse_timestamps": "true", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-12-06T07:57:41.952 INFO:teuthology.orchestra.run.smithi084.stdout: "log_flush_on_exit": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_host": "127.0.0.1", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_port": "12201", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_new": "1000", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_recent": "10000", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stderr_prefix": "", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stop_at_utilization": "0.970000", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_file": "true", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_graylog": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_journald": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_stderr": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_syslog": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "mandatory_topic_permissions": "false", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "max_rotating_auth_attempts": "10", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-12-06T07:57:41.953 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_action_on_write_error": "1", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_alternate_name_max": "8192", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_asio_thread_count": "2", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_export_pin": "true", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_dirs": "true", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_interval": "5", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_size_max": "100000", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_idle_threshold": "0.000000", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_interval": "10", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max": "-1", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max_until": "-1", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_merge_size": "50", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_midchunk": "0.300000", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_rebalance": "0.100000", 2023-12-06T07:57:41.954 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_start": "0.200000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_minchunk": "0.001000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_mode": "0", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_max": "1.200000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_min": "0.800000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_overload_epochs": "2", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_sample_interval": "3.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_bits": "3", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_rd": "25000.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_size": "10000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_wr": "10000.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_target_decay": "10.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_grace": "15.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_interval": "4.000000", 2023-12-06T07:57:41.955 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_mon_down_grace": "60", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_memory_limit": "4294967296", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_mid": "0.700000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_release_free_interval": "10", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_reservation": "0.050000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_interval": "1", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_threshold": "262144", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_delegate_inos_pct": "50", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_prealloc_inos": "1000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_connect_bootstrapping": "false", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_damage_table_max_entries": "10000", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-12-06T07:57:41.956 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_auth_pins": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_frag": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_scatterstat": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_subtrees": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_decay_halflife": "5.000000", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_default_dir_hash": "2", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_defer_session_stale": "true", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_deny_all_reconnect": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_keys_per_op": "16384", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_commit_size": "10", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_entries": "0", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_prefetch": "true", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dirstat_min_interval": "1.000000", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_after_rejoin": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_on_map": "false", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_file": "0", 2023-12-06T07:57:41.957 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_early_reply": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enable_op_tracker": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enforce_unique_name": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_forward_all_requests_to_auth": "false", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_cache_threshold": "1.500000", 2023-12-06T07:57:41.958 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_summarize_threshold": "10", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_grace": "15.000000", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_reset_grace": "1000", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_health_dummy": "false", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_migrator_session_race": "false", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_join_fs": "", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_journal_format": "1", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_create_at": "0", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_export_at": "0", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_import_at": "0", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_at": "0", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_expire_at": "0", 2023-12-06T07:57:41.959 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_replay_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_link_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_mdstable_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_openc_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_rename_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_shutdown_at": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_event_large_threshold": "524288", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_events_per_segment": "1024", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_major_segment_event_ratio": "12", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_events": "-1", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_segments": "128", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_pause": "false", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_segment_size": "0", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_corrupt_events": "false", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_unbounded_events": "false", 2023-12-06T07:57:41.960 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_warn_factor": "2.000000", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_caps_per_client": "1048576", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_flushes": "100000", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_requests": "100000", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_export_size": "20971520", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_file_recover": "32", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_files": "64", 2023-12-06T07:57:41.961 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops": "8192", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_snaps_per_dir": "100", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_metrics_update_interval": "2", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_per_client": "100", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_working_set": "10000", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_numa_node": "-1", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_complaint_time": "30.000000", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_duration": "600", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_size": "20", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_size": "20", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_threshold": "10", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_log_threshold": "5", 2023-12-06T07:57:41.962 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_grace": "15", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_interval": "5", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_caps": "30000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_threshold": "131072", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_threshold": "262144", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_reconnect_timeout": "45.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_interval": "1.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_gid": "0", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_uid": "0", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-12-06T07:57:41.963 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_evict": "true", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_timeout": "true", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_metadata_threshold": "16777216", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_shutdown_check": "0", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_skip_ino": "0", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sleep_rank_change": "0.000000", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_max_uid": "4294967294", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_min_uid": "0", 2023-12-06T07:57:41.964 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_rstat": "false", 2023-12-06T07:57:41.965 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_standby_replay_damaged": "false", 2023-12-06T07:57:41.965 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_symlink_recovery": "true", 2023-12-06T07:57:41.965 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_task_status_update_interval": "2.000000", 2023-12-06T07:57:41.965 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_exports": "0", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_fragments": "0", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_tick_interval": "5.000000", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_valgrind_exit": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_backtrace": "1", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_scatter": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_ino_prealloc": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_sessions": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mempool_debug": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_debug_omit_block_device_write": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_device_bytes": "1073741824", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_set": "false", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_size": "65536", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_bytes": "134217728", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_messages": "512", 2023-12-06T07:57:41.966 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_connect_retry_interval": "1.000000", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_disabled_modules": "", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_num_change": "128", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_bytes": "134217728", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_messages": "128", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_bytes": "134217728", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_messages": "128", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_bytes": "536870912", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_messages": "8192", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_pool": "true", 2023-12-06T07:57:41.967 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_service_beacon_grace": "60.000000", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_standby_modules": "true", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_period": "5", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_threshold": "5", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_tick_period": "2", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_accept_timeout_factor": "2.000000", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_delete": "true", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_size_one": "true", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_auth_validate_all_caps": "true", 2023-12-06T07:57:41.968 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_bytes": "104857600", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_directed_command_retry": "5", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval": "3.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_parallel": "3", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_log_interval": "1.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_interval": "10.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_timeout": "30.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_target_rank": "-1", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_allowed": "1.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-12-06T07:57:41.969 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file_level": "debug", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_file": "true", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog": "false", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-12-06T07:57:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_journald": "false", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_stderr": "false", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_bootstrap": "false", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_start": "false", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_trim": "true", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_persist_interval": "10", 2023-12-06T07:57:41.974 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_score_halflife": "43200", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_config_key_max_entry_size": "65536", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cpu_threads": "4", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_crush_min_required_version": "hammer", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_daemon_bytes": "419430400", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_crit": "5", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_warn": "5", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_size_warn": "16106127360", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_block_osdmap_trim": "false", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_json": "false", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_transactions": "false", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_extra_checks": "false", 2023-12-06T07:57:41.975 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_quincy": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_reef": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_delta_reset_interval": "10.000000", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_dns_srv_name": "ceph-mon", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_mkfs_grace": "120", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_default_strategy": "3", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_timeout": "5.000000", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_divisor": "2", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_timeout": "2.000000", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_enable_op_tracker": "true", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fake_pool_delete": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_force_quorum_join": "false", 2023-12-06T07:57:41.976 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_globalid_prealloc": "10000", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_detail_to_clog": "false", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_log_update_period": "5", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_max_detail": "50", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog": "true", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_interval": "600", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host": "172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790],172.21.15.107", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host_override": "", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_initial_members": "", 2023-12-06T07:57:41.977 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_keyvaluedb": "rocksdb", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease": "5.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_full_interval": "50", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max": "10000", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max_summary": "50", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_entries_per_event": "4096", 2023-12-06T07:57:41.979 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_epochs": "500", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mdsmap_epochs": "500", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mgrmap_epochs": "500", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_osd": "10000", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pg_per_osd": "10000", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pool_pg_num": "65536", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_force_trim_to": "0", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_skip_sanity": "false", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_autotune": "true", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_target": "2147483648", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_beacon_grace": "30", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_digest_period": "5", 2023-12-06T07:57:41.980 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_inactive_grace": "60", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_mkfs_grace": "240", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_min_osdmap_epochs": "50", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_complaint_time": "30", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_duration": "600", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_size": "20", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_size": "20", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_threshold": "10", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_log_threshold": "5", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-12-06T07:57:41.981 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_in": "false", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_new_in": "true", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size": "500", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size_min": "134217728", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_crush_smoke_test": "false", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_destroyed_out_interval": "600", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_interval": "600", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-12-06T07:57:41.982 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_force_trim_to": "0", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_full_ratio": "0.950000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_halflife": "3600", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_max_interval": "300", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_weight": "0.300000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_creating_pgs": "1024", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_initial_pgs": "1024", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_down_reporters": "2", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp": "true", 2023-12-06T07:57:41.983 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_report_timeout": "900", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_num_repaired": "10", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_op_age": "32.000000", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-12-06T07:57:41.984 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_interval": "2", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_min": "15", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_txsize": "2", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_stuck_threshold": "60", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_objects": "10000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_per_osd": "0", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_crit_threshold": "0", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_warn_threshold": "0", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_probe_timeout": "2.000000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_change": "0.050000", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_osds": "4", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-12-06T07:57:41.985 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_interval": "300", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_max_keys": "100", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_timeout": "300", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_session_timeout": "300", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_smart_report_timeout": "5", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stat_smooth_intervals": "6", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_min_size": "2", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_size": "4", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_subscribe_interval": "86400.000000", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_debug": "false", 2023-12-06T07:57:41.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_keys": "2000", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_size": "1048576", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_provider_kill_at": "0", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_requester_kill_at": "0", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_timeout": "60.000000", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_target_pg_per_osd": "100", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_tick_interval": "5", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_interval": "300.000000", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_use_min_delay_socket": "false", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_older_version_delay": "604800", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_filestore_osds": "false", 2023-12-06T07:57:41.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_misplaced": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_older_version": "true", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_app": "true", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_too_few_osds": "false", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-12-06T07:57:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": "", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_tracing_enabled": "false", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_op_threads": "3", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_buffer_size": "131072", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_cm": "false", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_device_name": "", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_dscp": "96", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_gid_idx": "0", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_local_gid": "", 2023-12-06T07:57:41.989 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_polling_us": "1000", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_port_num": "1", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_roce_ver": "1", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_send_buffers": "1024", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_sl": "3", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_support_srq": "true", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_type": "ib", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_reap_threshold": "5", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_before_connect": "false", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv4": "true", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv6": "false", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr1": "false", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr2": "true", 2023-12-06T07:57:41.990 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_max": "7568", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_min": "6800", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_prefer_ipv4": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_count": "3", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_delay": "5", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_client": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mds": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mgr": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mon": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_osd": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_client_mode": "crc secure", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_mode": "crc secure", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_type": "", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_compress_secure": "false", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_idle_timeout": "900", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_ready_timeout": "10", 2023-12-06T07:57:41.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_data": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_header": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bad_msg": "false", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bug": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_old_message": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_skipped_message": "false", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_unhandled_msg": "false", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_coremask": "0xF", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_devs_allowlist": "", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_enable_tso": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hugepages": "", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_flow_control": "true", 2023-12-06T07:57:41.992 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_lro": "true", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_memory_channel": "4", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_pmd": "", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_port_id": "0", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_corrupt_message_level": "1", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_on_send": "false", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_initial_backoff": "0.200000", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_max": "1.000000", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_probability": "0.000000", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_type": "", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_internal_delays": "0.000000", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_network_congestion": "0", 2023-12-06T07:57:41.993 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_socket_failures": "5000", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_learn_addr_from_peer": "true", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_accept_failures": "4", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_backoff": "15.000000", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_client_mode": "secure crc", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_cluster_mode": "secure crc", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_service_mode": "secure crc", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_min_size": "1024", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_mode": "none", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compression_algorithm": "snappy", 2023-12-06T07:57:41.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_min_cost": "65536", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_public_type": "", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_service_mode": "crc secure", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_listen_backlog": "512", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_nodelay": "true", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_rcvbuf": "0", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_type": "async", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "no_config_file": "false", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_completion_locks_per_session": "32", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_debug_inject_relock_delay": "false", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_op_bytes": "104857600", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_ops": "1024", 2023-12-06T07:57:41.995 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inject_no_watch_ping": "false", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_tick_interval": "5.000000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_timeout": "10.000000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_blackhole": "false", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "openssl_engine_opts": "", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_delay_time": "5.000000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_hist_halflife": "1000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_low_ops": "2", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_ops": "4", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_quantize_effort": "0.100000", 2023-12-06T07:57:41.996 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_slop": "0.020000", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_allow_recovery_below_min_size": "true", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_asio_thread_count": "2", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_async_recovery_min_cost": "100", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_auto_mark_unfound_lost": "false", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_retry_interval": "30.000000", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_max": "512", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_min": "64", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_degraded": "false", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_peering": "false", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_unfound": "true", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_beacon_report_interval": "300", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_duration": "30", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-12-06T07:57:41.997 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_max_block_size": "67108864", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_small_size_max_iops": "100", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blkin_trace_all": "false", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blocked_scrub_grace_period": "120", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_for_log_corruption": "false", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_update_on_start": "true", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_cap": "256", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_size_cap": "524288000", 2023-12-06T07:57:41.998 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_op_priority": "63", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_watch_timeout": "30", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_command_max_records": "256", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_compact_on_start": "false", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_chooseleaf_type": "0", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_initial_weight": "-1.000000", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_on_start": "true", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_weight_set": "true", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_duration": "0", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_feed_pullee": "-1", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-12-06T07:57:41.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_misdirected_ops": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_acting_change": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_purge_strays": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_op_order": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pg_log_writeout": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pretend_recovery_active": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_shutdown": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_trim_objects": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_cached_snaps": "false", 2023-12-06T07:57:42.000 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_missing_on_start": "false", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_snaps": "false", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_keys": "1024", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_stride": "524288", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_data_pool_replay_window": "5", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_notify_timeout": "30", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep": "0.000000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-12-06T07:57:42.001 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_discard_disconnected_ops": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_enable_op_tracker": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_extblkdev_plugins": "vdo", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_info": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_timeout": "15", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_function_tracing": "false", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_grace": "80", 2023-12-06T07:57:42.002 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_interval": "6", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_peers": "10", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_size": "2000", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_stale": "600", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_max_size": "100000", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_min_size": "1000", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_flush_on_shutdown": "true", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_size": "5120", 2023-12-06T07:57:42.003 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_kill_backfill_at": "0", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_loop_before_reset_tphandle": "64", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_cache_size": "50", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_dedup": "true", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max": "40", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max_bytes": "10485760", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_share_max_epochs": "40", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_name_len": "100", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_size": "0", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_backfills": "1", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_count": "5", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_period": "600", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_name_len": "2048", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_namespace_len": "256", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_size": "134217728", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-12-06T07:57:42.004 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_entries_per_request": "1024", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_blocked_by": "16", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_log_entries": "10000", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pgls": "1024", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_cost": "8388608", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_objects": "10", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_scrubs": "3", 2023-12-06T07:57:42.005 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_trimming_pgs": "2", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_op_reply_len": "64", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_size": "90", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_override_recovery_settings": "false", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_profile": "balanced", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-12-06T07:57:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_skip_benchmark": "false", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_base": "805306368", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_min": "134217728", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-12-06T07:57:42.007 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target": "4294967296", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_autotune": "false", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_pg_log_entries": "250", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_recovery_priority": "0", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_interval": "30", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_interval": "5", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_max_in_flight": "2", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_cache_shards": "32", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_op_tracker_shard": "32", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_auto_affinity": "true", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_node": "-1", 2023-12-06T07:57:42.008 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_prefer_iface": "true", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objecter_finishers": "1", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore": "bluestore", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_fuse": "false", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_tracing": "false", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_complaint_time": "30.000000", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_duration": "600", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_size": "20", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_size": "20", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_log_threshold": "5", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards": "0", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_hdd": "5", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_ssd": "8", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard": "0", 2023-12-06T07:57:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_min_cost": "65536", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue": "mclock_scheduler", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue_cut_off": "high", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_suicide_timeout": "150", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_timeout": "15", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_open_classes_on_start": "true", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_os_flags": "0", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_peering_op_priority": "255", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_cost": "1048576", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_priority": "5", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_dups_tracked": "3000", 2023-12-06T07:57:42.010 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_max": "10000", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_min": "100", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_object_context_cache_count": "64", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max": "500", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crimson": "false", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crush_rule": "-1", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_ec_fast_read": "false", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_bulk": "false", 2023-12-06T07:57:42.011 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nodelete": "false", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flags": "0", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_min_size": "0", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_num": "32", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pgp_num": "0", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_size": "2", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_type": "replicated", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_use_gmt_hitset": "true", 2023-12-06T07:57:42.012 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_push_per_object_cost": "1000", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_read_ec_check_for_errors": "false", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap": "true", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap_limit": "10", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_cost": "20971520", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_delay_start": "0.000000", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active": "0", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_hdd": "3", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_ssd": "10", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_chunk": "8388608", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_single_start": "1", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_priority": "3", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_warn_multiple": "16", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_priority": "5", 2023-12-06T07:57:42.013 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_retry_interval": "30.000000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep": "0.000000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_repair_during_recovery": "false", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_requested_scrub_priority": "120", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rollback_to_cluster_snap": "", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair": "false", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_hour": "0", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_week_day": "0", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_max": "25", 2023-12-06T07:57:42.014 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_min": "5", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_cost": "52428800", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_during_recovery": "false", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_hour": "0", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_week_day": "0", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_event_cost": "4096", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_invalid_stats": "true", 2023-12-06T07:57:42.015 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_load_threshold": "0.500000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_interval": "604800.000000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_preemptions": "5", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_min_interval": "86400.000000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_priority": "5", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_reservation_timeout": "300000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_sleep": "0.000000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_slow_reservation_response": "30000", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shutdown_pgref_assert": "false", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_data_digest": "false", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_smart_report_timeout": "5", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_cost": "1048576", 2023-12-06T07:57:42.016 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_priority": "5", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep": "0.000000", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_scrubbing": "15", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_transaction_size": "30", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-12-06T07:57:42.017 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_mode": "writeback", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tracing": "false", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_use_stale_snap": "false", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "osdc_blkin_trace_all": "false", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_kill_at": "0", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_max_join_drift": "10", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min": "500", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min_wait": "0.050000", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_propose_interval": "1.000000", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max": "500", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max_multiplier": "20", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_min": "10", 2023-12-06T07:57:42.018 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_stash_full_interval": "25", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_max": "500", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_min": "250", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "perf": "true", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_dir": "/usr/lib64/ceph", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "-", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrv": "", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "public_bind_addr": "-", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network": "", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network_interface": "", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_enabled": "false", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_session_max_number": "256", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_mon_op_timeout": "0", 2023-12-06T07:57:42.019 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_osd_op_timeout": "0", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_tracing": "false", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_atime_update_interval": "60", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_parent_reads": "false", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_snap_reads": "false", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blkin_trace_all": "false", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_expire_seconds": "0", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_on_break_lock": "true", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache": "true", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_block_writes_upfront": "false", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty": "25165824", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_object": "0", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_policy": "writearound", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_size": "33554432", 2023-12-06T07:57:42.020 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_target_dirty": "16777216", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_clone_copy_on_read": "false", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_compression_hint": "none", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_concurrent_management_ops": "10", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_clone_format": "auto", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_data_pool": "", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_features": "61", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_format": "2", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_map_options": "", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_order": "22", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_pool": "rbd", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_count": "0", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_unit": "0", 2023-12-06T07:57:42.021 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_disable_zero_copy_writes": "true", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_granularity_bytes": "65536", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_enable_alloc_hint": "true", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler": "simple", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_commit_age": "5.000000", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_interval": "0", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-12-06T07:57:42.022 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_order": "24", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_pool": "", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_splay_width": "4", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_parent_reads": "false", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_snap_reads": "false", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_type": "simple", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-12-06T07:57:42.023 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_autotune": "true", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_base": "805306368", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_target": "4294967296", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-12-06T07:57:42.024 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_delete_delay": "0", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_replay_delay": "0", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove": "false", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mtime_update_interval": "60", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_non_blocking_aio": "true", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_thread_timeout": "60", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_threads": "1", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_parent_cache_enabled": "false", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_mode": "disabled", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_path": "/tmp", 2023-12-06T07:57:42.025 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_size": "1073741824", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_plugins": "", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst": "0", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_limit": "0", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_exclude_ops": "0", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst": "0", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-12-06T07:57:42.026 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_limit": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_limit": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_limit": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_schedule_tick_min": "50", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_limit": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_limit": "0", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_quiesce_notification_attempts": "10", 2023-12-06T07:57:42.027 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_read_from_replica_policy": "default", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_max_bytes": "524288", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_trigger_requests": "10", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_request_timed_out_seconds": "30", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_skip_partial_discard": "true", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_tracing": "false", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_names": "true", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_pool": "true", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_base_url": "", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_log_level": "", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_acl_grants_max_num": "100", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_admin_entry": "admin", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_backend_store": "rados", 2023-12-06T07:57:42.028 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_barbican_url": "", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_beast_enable_async": "true", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache": "false", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache_size": "10000", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_max_aio": "128", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_ttl": "600", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_enabled": "true", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_expiry_interval": "900", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_lru_size": "10000", 2023-12-06T07:57:42.029 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_config_store": "rados", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_content_length_compat": "false", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress": "true", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cors_rules_max_num": "100", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cross_domain_policy": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_default_encryption_key": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_addr": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_ca_path": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_cert": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_key": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_password": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_username": "", 2023-12-06T07:57:42.030 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_require_ssl": "true", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_suppress_logs": "true", 2023-12-06T07:57:42.031 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_addr": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_auth": "token", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_namespace": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_prefix": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_token_file": "", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_buffersize": "524288", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_limit": "1024", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_time": "300", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_tcp_keepalive": "0", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-12-06T07:57:42.032 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_fadvise": "4", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_host": "127.0.0.1", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_port": "6379", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_changes_size": "1000", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_num_shards": "128", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_window": "30", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_notify_interval_msec": "0", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_poll_interval": "20", 2023-12-06T07:57:42.033 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_spawn_window": "20", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_data_log_backing": "fifo", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_region_info_oid": "default.region", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_defer_to_bucket_acls": "", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-12-06T07:57:42.034 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_res": "500.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_name": "", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_s3website_name": "", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding": "true", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_gc_threads": "true", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_lc_threads": "true", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_ops_log": "false", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_quota_threads": "true", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_static_website": "false", 2023-12-06T07:57:42.035 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_usage_log": "false", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enforce_swift_acls": "true", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_exit_timeout_secs": "120", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_expose_bucket": "false", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_extended_http_attrs": "", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_filter": "none", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontends": "beast port=7480", 2023-12-06T07:57:42.036 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_concurrent_io": "10", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred": "50", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_objs": "32", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_queue_size": "134213632", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_trim_chunk": "16", 2023-12-06T07:57:42.037 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_obj_min_wait": "7200", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_max_time": "3600", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_period": "3600", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_window_size": "16777216", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_healthcheck_disabling_path": "", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ignore_get_invalid_range": "false", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_init_timeout": "300", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-12-06T07:57:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_domain": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password_path": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_project": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_tenant": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token_path": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_user": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_api_version": "2", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_domain": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_password": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_project": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_tenant": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_user": "", 2023-12-06T07:57:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_implicit_tenants": "false", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_enabled": "false", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_token_cache_size": "10000", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_url": "", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_verify_ssl": "true", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_debug_interval": "-1", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_lock_max_time": "90", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_objs": "32", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_rules": "1000", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_worker": "3", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_wp_worker": "3", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_thread_delay": "0", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_dnattr": "uid", 2023-12-06T07:57:42.040 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchfilter": "", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_uri": "ldaps://", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_http_headers": "", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_nonexistent_bucket": "false", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name_utc": "false", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_name_len": "0", 2023-12-06T07:57:42.041 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_size": "0", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attrs_num_in_req": "0", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_chunk_size": "4194304", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_concurrent_requests": "1024", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_dynamic_shards": "1999", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_listing_results": "1000", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_notify_retries": "10", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_objs_per_shard": "100000", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_param_size": "1048576", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_size": "5368709120", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_slo_entries": "1000", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_log_max_shards": "64", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_notify_interval_msec": "200", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_poll_interval": "20", 2023-12-06T07:57:42.042 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_spawn_window": "20", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mp_lock_max_time": "600", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_min_part_size": "5242880", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_size": "2017", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lanes": "5", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_max_gc": "300", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_gc_threads": "false", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_lc_threads": "false", 2023-12-06T07:57:42.043 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_quota_threads": "false", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_sync_thread": "false", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_async_rados_threads": "32", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_control_oids": "8", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_numa_node": "-1", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_stripe_size": "4194304", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_chunk_size": "100", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_gc_interval": "600", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_hints_num_shards": "127", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_timeout": "600", 2023-12-06T07:57:42.044 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_tracing": "false", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_token": "", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_url": "", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_verify_ssl": "true", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_rados": "false", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_socket_path": "", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval": "2.000000", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval_max": "30.000000", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_root_pool": ".rgw.root", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-12-06T07:57:42.045 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lanes": "3", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_max_buckets": "100", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_partitions": "3", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_continue": "true", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_prohibited_content_length": "false", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_tracing": "false", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm": "", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_id": "", 2023-12-06T07:57:42.046 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region": "", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region_root_pool": ".rgw.root", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_region_enforcement": "false", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_request_uri": "", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_batch_size": "64", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_max_aio": "128", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_num_logs": "16", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_thread_interval": "600", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_resolve_cname": "false", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rest_getusage_op_compat": "false", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_run_sync_thread": "true", 2023-12-06T07:57:42.047 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_keystone": "false", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_ldap": "false", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_rados": "true", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_sts": "false", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_success_create_obj_status": "0", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_scheduler_type": "throttler", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_script_uri": "", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_service_provider_name": "", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_id": "", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_secret": "", 2023-12-06T07:57:42.048 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_entry": "sts", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_key": "sts", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_max_session_duration": "43200", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_min_session_duration": "900", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_token_introspection_url": "", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_account_in_url": "false", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_entry": "auth", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_url": "", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_custom_header": "", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_enforce_content_length": "false", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_need_stats": "true", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_tenant_name": "", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_token_expiration": "86400", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url": "", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url_prefix": "swift", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_versioning_enabled": "false", 2023-12-06T07:57:42.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_lease_period": "120", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_interval": "1200", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_obj_etag_verify": "false", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_history_size": "4096", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_thread_pool_size": "512", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_max_retries": "0", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-12-06T07:57:42.050 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_comment": "", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_createby": "", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_encoding": "", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_flag": "false", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_max_size": "5368709120", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_origin": "", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_sha_unit": "524288", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_tracker": "", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_trust_forwarded_https": "false", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_tick_interval": "30", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_shards": "32", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_user_shards": "1", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_use_opa_authz": "false", 2023-12-06T07:57:42.051 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache": "false", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache_size": "10000", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_size": "-1", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_max_buckets": "1000", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_policies_max_num": "100", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_interval": "86400", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_unique_email": "true", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_verify_ssl": "true", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_website_routing_rules_max_num": "50", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone": "", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_id": "", 2023-12-06T07:57:42.052 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup": "", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_id": "", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_auto_session_clear": "true", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_skip_bucket_step": "false", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_block_size": "4096", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_shard_bits": "4", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_size": "536870912", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_type": "binned_lru", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-12-06T07:57:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_compaction_stats": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_extended_stats": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_memory_stats": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_index_type": "binary_search", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_log_to_ceph_log": "true", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_metadata_block_size": "4096", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_partition_filters": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_perf": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-12-06T07:57:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_renewal_timeout": "10", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "run_dir": "/var/run/ceph", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_block_create": "true", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cache_lru_size": "67108864", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cbjournal_size": "5368709120", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_data_delta_based_overwrite": "0", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_max_object_size": "16777216", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_device_size": "53687091200", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_capacity": "16", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_iodepth_limit": "5", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_main_device_type": "SSD", 2023-12-06T07:57:42.055 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_concurrent_transactions": "8", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_segment_size": "67108864", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "set_keepcaps": "false", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "setgroup": "", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser": "", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser_match_path": "", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_misplaced_ratio": "0.050000", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "thp": "false", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_default_timeout": "60", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_empty_queue_max_wait": "2", 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout: "throttler_perf_counter": "true" 2023-12-06T07:57:42.056 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:42.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-12-06T07:57:42.317 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2023-12-06T07:57:42.317 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T07:57:42.413 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2023-12-06T07:57:42.413 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T07:57:42.508 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2023-12-06T07:57:42.509 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T07:57:42.518 DEBUG:teuthology.orchestra.run.smithi084:> type ceph-objectstore-tool 2023-12-06T07:57:42.532 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T07:57:42.532 DEBUG:teuthology.orchestra.run.smithi107:> type ceph-objectstore-tool 2023-12-06T07:57:42.547 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T07:57:42.548 INFO:teuthology.run_tasks:Running task mon_recovery... 2023-12-06T07:57:42.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T07:57:42.582 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-12-06T07:57:42.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:57:42.582 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:57:42.582 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2023-12-06T07:57:42.582 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T07:57:42.583 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T07:57:42.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2023-12-06T07:57:42.583 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-12-06T07:57:42.584 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-12-06T07:57:42.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:57:42.584 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-12-06T07:57:42.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:57:42.585 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-12-06T07:57:42.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T07:57:42.595 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:42.592+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:42.693 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:42.690+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:42.735 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:42.735 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:42.735 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:57:42.735 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:42.737 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:57:42.738 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T07:57:42.738 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:57:42.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:57:42.794 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:42.791+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:42.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:57:42.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:57:42.897 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:42.896+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:42.911 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:42.912 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:42.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:57:42.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:57:42.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:57:43.003 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:43.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:43.114 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:43.114 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:43.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:57:43.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:57:43.141 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:57:43.148 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:43.146+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:43.198 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:43.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:43.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:57:43.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:57:43.290 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:43.290 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:43.293 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:43.293+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:43.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:57:43.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:57:43.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:57:43.402 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:43.397+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:43.493 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:43.493 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:43.495 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:43.495+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:43.500 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:43.501 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":22,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T07:57:31.995910+0000","last_up_change":"2023-12-06T07:57:18.326272+0000","last_in_change":"2023-12-06T07:56:37.562649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":933623275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":933623275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":933623275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":933623275}]},"public_addr":"172.21.15.107:6809/933623275","cluster_addr":"172.21.15.107:6810/933623275","heartbeat_back_addr":"172.21.15.107:6812/933623275","heartbeat_front_addr":"172.21.15.107:6811/933623275","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T07:57:43.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:57:43.515 INFO:tasks.mon_recovery.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-06T07:57:43.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-06T07:57:43.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:57:43.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:57:43.596 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:43.596+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:43.648 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:57:43.648 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:57:43.667 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:43.667 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:43.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:57:43.717 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:43.717+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:43.789 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:57:43.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:57:43.798 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:43.798+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:43.856 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:43.856 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:43.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:57:43.900 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:43.899+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:44.001 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:44.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:44.016 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.016 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:57:44.102 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:44.102+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:44.157 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.157 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:57:44.173 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2023-12-06T07:57:44.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-12-06T07:57:44.228 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:44.228+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:44.304 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:44.304+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:44.321 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.321 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:57:44.412 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:44.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:44.497 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.497 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:57:44.506 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:44.506+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:44.607 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:44.606+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:44.690 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.690 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:57:44.722 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T07:57:44.743 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:44.743+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:44.809 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:44.809+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:44.838 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:44.838 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:44.840 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 8 2023-12-06T07:57:44.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:57:44.852 INFO:tasks.mon_recovery:mon ids = ['a', 'c', 'b'] 2023-12-06T07:57:44.853 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T07:57:44.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:57:44.933 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:44.932+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:45.011 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:45.010+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:45.024 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.025 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:57:45.112 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:45.110+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:45.203 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.204 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:57:45.213 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:45.212+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:45.313 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:45.313+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:45.402 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.402 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:57:45.414 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:45.413+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:45.494 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":73,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:57:45.507 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T07:57:45.508 INFO:tasks.mon_recovery:verifying all monitors are in the quorum 2023-12-06T07:57:45.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-06T07:57:45.550 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:45.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:45.581 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.581 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:57:45.635 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:45.635+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:45.717 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:45.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:45.722 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.722 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:57:45.756 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "a", 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 0, 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "leader", 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "election_epoch": 14, 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum": [ 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_age": 73, 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "required_con": "2449958758054445060", 2023-12-06T07:57:45.757 INFO:teuthology.orchestra.run.smithi084.stdout: "required_mon": [ 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_con": "4540138322906710015", 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_mon": [ 2023-12-06T07:57:45.758 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "outside_quorum": [], 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "extra_probe_peers": [], 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "sync_provider": [], 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": { 2023-12-06T07:57:45.759 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 1, 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release": 18, 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release_name": "reef", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "election_strategy": 3, 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "disallowed_leaders: ": "", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false, 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "tiebreaker_mon": "", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_ranks: ": "", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "persistent": [ 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.760 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "optional": [] 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "mons": [ 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 0, 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "a", 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3300", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789/0", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6789/0", 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.762 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 1, 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "b", 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:3300", 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.763 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789", 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789/0", 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6789/0", 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 2, 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "c", 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3301", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790/0", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6790/0", 2023-12-06T07:57:45.765 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "feature_map": { 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "mon": [ 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": [ 2023-12-06T07:57:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client": [ 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 4 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": [ 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.767 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false 2023-12-06T07:57:45.768 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:45.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-06T07:57:45.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:57:45.818 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:45.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:45.902 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:45.902 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:45.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:57:45.953 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:45.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "c", 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 2, 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "peon", 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: "election_epoch": 14, 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum": [ 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T07:57:45.989 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_age": 73, 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "required_con": "2449958758054445060", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "required_mon": [ 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.990 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_con": "4540138322906710015", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_mon": [ 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "outside_quorum": [], 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "extra_probe_peers": [], 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "sync_provider": [], 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": { 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 1, 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release": 18, 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release_name": "reef", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "election_strategy": 3, 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "disallowed_leaders: ": "", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false, 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "tiebreaker_mon": "", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_ranks: ": "", 2023-12-06T07:57:45.992 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "persistent": [ 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "optional": [] 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: "mons": [ 2023-12-06T07:57:45.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 0, 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "a", 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3300", 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789", 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.994 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789/0", 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6789/0", 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 1, 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "b", 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:3300", 2023-12-06T07:57:45.995 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789", 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789/0", 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6789/0", 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 2, 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "c", 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3301", 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790", 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790/0", 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6790/0", 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "feature_map": { 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "mon": [ 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:45.998 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": [ 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 2 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "client": [ 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 3 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.000 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false 2023-12-06T07:57:46.000 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:46.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-06T07:57:46.020 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:46.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:46.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:57:46.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:57:46.082 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:46.082 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:46.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:57:46.133 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:46.132+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:46.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:57:46.163 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:57:46.174 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "b", 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 1, 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "peon", 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "election_epoch": 14, 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum": [ 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_age": 74, 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "required_con": "2449958758054445060", 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "required_mon": [ 2023-12-06T07:57:46.175 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_con": "4540138322906710015", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "quorum_mon": [ 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:46.176 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "outside_quorum": [], 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "extra_probe_peers": [], 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "sync_provider": [], 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": { 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 1, 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:57:46.177 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:55:36.076779Z", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release": 18, 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "min_mon_release_name": "reef", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "election_strategy": 3, 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "disallowed_leaders: ": "", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false, 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "tiebreaker_mon": "", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_ranks: ": "", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "features": { 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "persistent": [ 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "kraken", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "luminous", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "mimic", 2023-12-06T07:57:46.178 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap-prune", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "nautilus", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "octopus", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "pacific", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "elector-pinging", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "quincy", 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "reef" 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "optional": [] 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "mons": [ 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.179 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 0, 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "a", 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3300", 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789", 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6789/0", 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6789/0", 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 1, 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "b", 2023-12-06T07:57:46.181 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:3300", 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789", 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6789/0", 2023-12-06T07:57:46.182 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6789/0", 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "rank": 2, 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "c", 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:3301", 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790", 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 0 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6790/0", 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6790/0", 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "priority": 0, 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "{}" 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.184 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.185 INFO:teuthology.orchestra.run.smithi084.stdout: "feature_map": { 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "mon": [ 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": [ 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 5 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.186 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client": [ 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": [ 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:46.187 INFO:teuthology.orchestra.run.smithi084.stdout: "features": "0x3f01cfbffffdffff", 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: "release": "luminous", 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 1 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": false 2023-12-06T07:57:46.188 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:46.189 INFO:tasks.mon_recovery:restarting each monitor in turn 2023-12-06T07:57:46.190 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T07:57:46.190 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:57:46.204 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T07:57:46.193+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86655) UID: 0 2023-12-06T07:57:46.205 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T07:57:46.194+0000 9f54640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T07:57:46.221 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:46.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:46.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:57:46.260 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:57:46.265 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:46.265 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:46.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:57:46.360 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:46.359+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:46.362 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:57:46.362 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:57:46.441 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:46.440+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:46.459 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:46.459 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:46.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:57:46.461 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:57:46.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:57:46.524 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:46.524+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:46.548 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:57:46.549 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:57:46.625 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:46.625+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:46.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:57:46.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:57:46.645 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:46.646 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:46.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:57:46.727 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:46.725+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:46.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:57:46.734 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:57:46.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T07:57:46.828 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:46.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:46.844 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.2536179360000004, 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:57:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:57:46.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:57:46.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:57:46.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:57:46.849 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:57:46.849 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:57:46.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:57:46.911 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:37.990 86686== 2023-12-06T07:57:46.911 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:37.990 86686== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:57:46.922 INFO:tasks.ceph.mon.a.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:57:46.928 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:46.927+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:46.987 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:46.987 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:46.987 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:57:46.987 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:46.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:57:47.029 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:47.028+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:47.130 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:47.130 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:47.130 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:57:47.130 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:57:47.130 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:47.131 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:47.129+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:47.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:57:47.232 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:47.231+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:47.314 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:47.314 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:57:47.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:57:47.315 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:57:47.315 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:47.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:57:47.365 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:47.364+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:47.433 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:47.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:47.496 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:57:47.497 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:47.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:57:47.534 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:47.534+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:47.635 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:47.634+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:47.644 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:47.644 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:47.644 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:57:47.644 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:57:47.644 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:47.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:57:47.738 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:47.737+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:47.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:57:47.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:57:47.749 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T07:57:47.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T07:57:47.837 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:47.837+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.914 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.750367035, 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.915 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:57:47.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:57:47.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:57:47.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.279253581999999, 2023-12-06T07:57:47.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:57:47.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:57:47.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:57:47.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:57:47.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:57:47.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:57:47.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.049061203000001, 2023-12-06T07:57:47.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:57:47.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.932 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.933 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:57:47.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:57:47.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:57:47.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:57:47.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:57:47.940 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.048353654, 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:57:47.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:57:47.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.946 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.047716659999999, 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.947 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:57:47.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:57:47.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.990651464999999, 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.953 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:57:47.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.959 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.959 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:57:47.959 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.989160693999999, 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.960 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:57:47.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:57:47.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:57:47.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.956811032000001, 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.968 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:57:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.950180257, 2023-12-06T07:57:47.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:57:47.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:57:47.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:57:47.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:57:47.979 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.949529576, 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.946056885000001, 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.985 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:57:47.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:57:47.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:57:47.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.944918163000001, 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.990 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:47.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:47.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.943970466, 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:47.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:57:47.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:57:47.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:57:48.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.915606472, 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:57:48.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.887396955, 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:57:48.006 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:57:48.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:57:48.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:57:48.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.277199021000001, 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.012 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:57:48.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.276554181000002, 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.017 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:57:48.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.256087427000001, 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:57:48.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.255338511000001, 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:57:48.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:57:48.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:57:48.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.254696442, 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:57:48.033 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:57:48.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:57:48.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.038 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:57:48.040 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:48.039+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:48.140 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:48.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:48.188 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.188 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:48.188 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:57:48.188 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:57:48.282 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:48.280+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:48.341 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:48.340+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 23, 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:57:44.280608+0000", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2023-12-06T07:57:48.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T07:57:48.367 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "jewel", 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:57:48.368 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:57:48.369 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:57:48.370 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:57:48.371 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T07:57:48.372 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:57:48.373 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.374 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T07:57:48.374 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T07:57:48.374 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:57:48.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:57:48.377 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:57:48.378 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 4.9800000190734863, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 4.9800000190734863, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 4.380000114440918, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 4.380000114440918, 2023-12-06T07:57:48.379 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T07:57:48.383 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 16, 2023-12-06T07:57:48.384 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.385 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.386 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.387 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.388 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:57:48.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.391 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.393 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.395 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.396 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.397 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T07:57:48.398 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T07:57:48.399 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:57:48.400 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.401 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T07:57:48.402 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T07:57:48.403 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.404 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:57:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T07:57:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T07:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.409 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:57:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T07:57:48.411 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.412 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:57:48.413 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.414 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.415 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T07:57:48.416 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2023-12-06T07:57:48.417 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:48.419 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.421 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.421 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:48.421 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:57:48.421 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:57:48.421 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:57:48.422 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T07:57:44.280608+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T07:57:48.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:57:48.443 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:48.443+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:48.544 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:48.543+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:48.597 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.597 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:57:48.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:57:48.597 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:57:48.597 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:57:48.651 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:48.650+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:48.746 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:48.746+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:48.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:57:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.787 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:48.787 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:57:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:57:48.881 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:48.880+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:48.948 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:48.946+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:48.982 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:48.982 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:57:48.982 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:57:48.982 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:57:48.982 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:57:49.026 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:57:49.027 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:57:49.049 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:49.049+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:49.078 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:57:49.079 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:57:49.090 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-12-06T07:57:47.804729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218460,"kb_used_data":2296,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756260,"statfs":{"total":773094113280,"available":772870410240,"internally_reserved":0,"allocated":2351104,"data_stored":1327898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001827"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.284168+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:57:33.284168+0000","last_peered":"2023-12-06T07:57:33.284168+0000","last_clean":"2023-12-06T07:57:33.284168+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:57:33.284168+0000","last_undegraded":"2023-12-06T07:57:33.284168+0000","last_fullsized":"2023-12-06T07:57:33.284168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.266287+0000","last_change":"2023-12-06T07:57:30.401832+0000","last_active":"2023-12-06T07:57:33.266287+0000","last_peered":"2023-12-06T07:57:33.266287+0000","last_clean":"2023-12-06T07:57:33.266287+0000","last_became_active":"2023-12-06T07:57:30.374582+0000","last_became_peered":"2023-12-06T07:57:30.374582+0000","last_unstale":"2023-12-06T07:57:33.266287+0000","last_undegraded":"2023-12-06T07:57:33.266287+0000","last_fullsized":"2023-12-06T07:57:33.266287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:20.205766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.288209+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:57:33.288209+0000","last_peered":"2023-12-06T07:57:33.288209+0000","last_clean":"2023-12-06T07:57:33.288209+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:57:33.288209+0000","last_undegraded":"2023-12-06T07:57:33.288209+0000","last_fullsized":"2023-12-06T07:57:33.288209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.267983+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:57:33.267983+0000","last_peered":"2023-12-06T07:57:33.267983+0000","last_clean":"2023-12-06T07:57:33.267983+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:57:33.267983+0000","last_undegraded":"2023-12-06T07:57:33.267983+0000","last_fullsized":"2023-12-06T07:57:33.267983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.274556+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:57:33.274556+0000","last_peered":"2023-12-06T07:57:33.274556+0000","last_clean":"2023-12-06T07:57:33.274556+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:57:33.274556+0000","last_undegraded":"2023-12-06T07:57:33.274556+0000","last_fullsized":"2023-12-06T07:57:33.274556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.228268+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:57:32.228268+0000","last_peered":"2023-12-06T07:57:32.228268+0000","last_clean":"2023-12-06T07:57:32.228268+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:57:32.228268+0000","last_undegraded":"2023-12-06T07:57:32.228268+0000","last_fullsized":"2023-12-06T07:57:32.228268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.266918+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:57:33.266918+0000","last_peered":"2023-12-06T07:57:33.266918+0000","last_clean":"2023-12-06T07:57:33.266918+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:57:33.266918+0000","last_undegraded":"2023-12-06T07:57:33.266918+0000","last_fullsized":"2023-12-06T07:57:33.266918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:33.278736+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:57:33.278736+0000","last_peered":"2023-12-06T07:57:33.278736+0000","last_clean":"2023-12-06T07:57:33.278736+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:57:33.278736+0000","last_undegraded":"2023-12-06T07:57:33.278736+0000","last_fullsized":"2023-12-06T07:57:33.278736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":119,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-06T07:57:32.232699+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:57:32.232699+0000","last_peered":"2023-12-06T07:57:32.232699+0000","last_clean":"2023-12-06T07:57:32.232699+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:57:32.232699+0000","last_undegraded":"2023-12-06T07:57:32.232699+0000","last_fullsized":"2023-12-06T07:57:32.232699+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7439999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.167999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8989999999999991}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.888999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.850999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4080000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.647}]}]},{"osd":2,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.071999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.466999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6470000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.245000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.630000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.116}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.623000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":461686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8059999999999992}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":100.703}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.769}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.071000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.734999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.109999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476746,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":63846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5980000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.096}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.125999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.417999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.702}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.402999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5019999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.27}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.984999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.359}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.66}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.720000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.242}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.438000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.063000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.602}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.140999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":465976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3239999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9450000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.471}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.444000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.102}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.672000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.478000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.422000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.73}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.132000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.242000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:57:49.092 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [7, 2] 2023-12-06T07:57:49.092 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.6', '7', '2'] 2023-12-06T07:57:49.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.6 7 2 2023-12-06T07:57:49.117 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:57:49.117 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:57:49.150 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:49.150+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:49.173 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:57:49.173 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.7451184770000001, 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:57:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:57:49.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:57:49.178 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:57:49.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:57:49.207 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:57:49.207 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:57:49.251 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:49.250+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:49.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:57:49.297 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:57:49.333 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:57:49.333 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:57:49.333 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:57:49.333 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:57:49.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:57:49.352 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:49.351+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:49.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:57:49.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:57:49.454 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:49.452+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:49.472 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:49.472 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:49.478 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:57:49.478 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:57:49.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:57:49.555 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:49.554+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:49.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:57:49.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:57:49.612 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:49.613 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:49.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:57:49.655 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:49.654+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:49.659 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:57:49.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:57:49.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T07:57:49.756 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:49.756+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:49.791 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:49.791 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:49.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:57:49.859 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:49.858+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:49.927 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:49.927 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:49.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:57:49.960 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:49.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:50.062 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:50.061+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:50.068 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.068 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:57:50.163 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:50.162+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:50.244 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.244 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:57:50.264 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:50.263+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:50.365 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:50.365+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:50.379 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.379 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:57:50.466 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:50.465+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:50.556 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.556 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:57:50.567 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:50.566+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:50.678 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:50.679+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:50.693 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.693 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:57:50.772 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:50.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:50.873 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:50.872+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:50.875 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:50.875 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:50.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:57:50.973 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:50.974+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:51.012 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.012 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:57:51.074 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:51.073+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:51.155 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.155 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:57:51.176 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:51.176+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:51.277 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:51.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:51.335 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.335 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:57:51.377 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:51.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:51.478 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:51.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:51.512 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.512 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:57:51.579 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:51.579+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:51.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:57:51.691 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.691 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.691 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:51.690+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:51.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:57:51.781 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:51.781+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:51.877 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:51.877 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:51.881 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:51.882+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:51.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:57:51.982 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:51.982+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:52.015 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.015 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:57:52.083 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:52.084+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:52.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.153 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:57:52.185 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:52.185+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:52.289 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:52.289+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:52.291 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.291 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:57:52.291 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T07:57:52.292 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T07:57:52.293 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T07:57:52.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:57:52.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:57:52.389 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:52.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:52.490 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:52.491+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:52.498 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.498 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:57:52.594 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:52.594+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:52.680 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.680 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:57:52.691 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:52.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:52.793 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:52.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:52.863 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:52.864 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:52.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:57:52.894 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:52.894+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:52.966 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":80,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:57:52.995 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:52.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:53.007 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.007 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:57:53.097 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:53.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:53.157 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.157 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:57:53.209 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:53.209+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:53.299 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:53.300+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:53.345 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.346 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:57:53.400 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:53.400+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:53.502 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:53.502+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:53.523 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.523 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:57:53.616 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:53.616+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:53.705 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.705 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.706 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:53.706+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:53.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:57:53.806 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:53.806+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:53.883 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:53.883 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:53.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:57:53.935 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:53.936+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:54.008 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:54.009+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:54.020 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.021 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:57:54.110 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:54.110+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:54.199 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.199 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:57:54.210 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:54.210+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:54.311 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:54.311+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:54.335 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.335 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:57:54.413 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:54.413+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:54.473 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.473 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:57:54.513 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:54.513+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:54.614 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:54.614+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:54.650 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.651 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:57:54.715 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:54.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:54.789 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.789 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:57:54.816 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:54.816+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:54.916 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:54.917+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:54.968 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:54.968 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:54.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:57:55.022 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:55.022+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:55.107 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.108 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:57:55.118 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:55.118+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:55.219 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:55.218+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:55.285 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.285 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:57:55.336 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:55.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:55.420 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:55.421+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:55.479 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.479 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:57:55.533 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:55.533+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:55.616 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.616 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.622 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:55.623+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:55.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:57:55.723 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:55.724+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:55.796 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.796 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:57:55.849 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:55.849+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:55.925 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:55.925+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:55.933 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:55.933 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:55.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T07:57:55.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:57:55.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:57:56.027 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:56.027+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:56.116 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.116 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:57:56.127 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:56.127+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:56.227 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:56.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:56.297 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.298 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:57:56.329 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:56.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:56.430 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:56.430+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:56.477 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.477 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:57:56.530 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:56.531+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:56.631 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:56.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:56.656 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.656 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:57:56.732 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:56.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:56.793 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.794 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:57:56.845 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:56.844+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:56.934 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:56.934+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:56.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:57:57.035 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:57.035+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:57.123 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.123 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:57:57.135 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:57.136+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:57.237 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:57:57.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:57:57.260 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.260 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:57:57.339 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:57.339+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:57.394 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.394 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:57:57.439 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:57.439+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:57.539 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:57.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:57.573 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.573 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:57:57.640 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:57.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:57.711 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.711 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:57:57.763 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:57.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:57.843 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:57.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:57.890 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:57.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:57:57.945 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:57.945+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:58.030 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.030 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:57:58.046 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:58.046+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:58.147 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:58.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:58.209 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.209 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:57:58.263 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:58.263+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:58.344 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.345 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.348 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:58.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:58.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:57:58.449 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:58.449+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:58.528 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.528 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:57:58.579 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:58.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:58.651 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:58.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:58.705 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.706 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:57:58.757 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:58.757+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:58.854 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:57:58.854+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:57:58.881 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:58.881 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:58.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:57:58.954 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:58.954+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:59.028 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.029 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:57:59.055 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:59.056+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:59.156 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:59.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:59.206 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.206 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:57:59.256 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:59.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:59.358 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:57:59.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:57:59.383 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.383 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:57:59.459 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:57:59.458+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:57:59.521 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.521 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:57:59.572 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:57:59.572+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:57:59.660 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:59.660+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:59.702 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.702 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:57:59.760 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:57:59.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:57:59.838 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.838 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:57:59.861 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:57:59.862+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:57:59.962 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:57:59.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:57:59.976 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:57:59.976 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:57:59.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:00.063 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:00.063+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:00.114 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.114 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:00.165 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:00.164+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:00.279 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:00.279+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:00.293 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.294 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:00.389 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:00.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:00.467 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:00.467+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:00.471 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.471 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:00.568 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:00.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:00.652 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.652 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:00.702 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:00.702+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:00.770 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:00.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:00.837 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.837 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:00.888 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:00.888+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:00.972 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:00.972+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:00.977 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:00.977 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:00.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:01.036 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:01.049 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T07:58:01.049 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T07:58:01.049 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T07:58:01.080 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:01.080+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:01.091 INFO:tasks.ceph.mon.a:Started 2023-12-06T07:58:01.091 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T07:58:01.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:01.120 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:01.120 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:01.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:01.174 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:01.174+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:01.185 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:01.185 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:01.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:01.289 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:01.306 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:01.306+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:01.317 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:01.317 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:01.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:01.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:01.397 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:01.397 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:01.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:01.477 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:01.478+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:01.495 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:01.495 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:01.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:01.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:01.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:01.578 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:01.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:01.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:01.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:01.638 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T07:58:01.697 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:01.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:01.713 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.6 pg_upmap mapping to [7,2] 2023-12-06T07:58:01.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:01.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:01.744 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:01.745 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:01.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:01.780 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:01.780+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:01.785 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:01.831 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:01.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:01.917 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:01.917+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:01.917 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:01.917 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.947498482999997, 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.283717482, 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.280436852999998, 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.975660237, 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.974817139999999, 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.952281634000002, 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.950289214999998, 2023-12-06T07:58:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.949756950000001, 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.932832040999998, 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.931995407000002, 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.907548138999999, 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.901645774999999, 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.896939779, 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.894637904, 2023-12-06T07:58:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.894139580999997, 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:02.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.893685669999996, 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:02.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:02.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.89246327, 2023-12-06T07:58:02.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.247297289000002, 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.022 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.239858873999999, 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:02.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:02.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:02.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:02.027 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.229785501999999, 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:02.031 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:02.033 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:02.033+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:02.134 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:02.133+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:02.212 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:02.213 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:02.213 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:02.213 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:02.235 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:02.235+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:02.336 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:02.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:02.370 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:02.370 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:02.370 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:02.370 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:02.371 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:02.437 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:02.437+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:02.517 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:02.517 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:02.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:02.517 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:02.518 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:02.539 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:02.538+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:02.640 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:02.640+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:02.657 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:02.657 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:02.657 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:02.657 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:02.752 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:02.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:02.841 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:02.841+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:02.851 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:02.851 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:02.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:02.942 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:02.942+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:03.032 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:03.032 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 22.455714408999999, 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:03.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:03.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:03.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:03.036 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:03.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:03.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:03.037 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:03.037 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:03.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:03.043 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:03.043+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:03.144 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:03.144+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:03.245 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:03.245+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:03.332 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:03.332 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:03.332 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:03.332 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:03.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:03.345 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:03.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:03.446 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:03.446+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:03.470 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:03.471 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:03.471 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:03.471 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:03.471 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:03.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:03.534 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:54.582 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.535 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:54.584 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.547 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:03.547+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:03.576 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:54.624 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.582 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.661 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.591 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.670 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.647 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:54.727 86690== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:03.648 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:03.647+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:03.652 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:03.652 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:03.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:03.652 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:03.652 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:03.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T07:58:03.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:03.749 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:03.749+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:03.851 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:03.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:03.854 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:03.854 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:03.854 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:03.854 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:03.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:03.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:03.993 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:03.992+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:04.053 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:04.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:04.053 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:04.053 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:04.053 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:04.053 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:04.053 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:04.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:04.155 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:04.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:04.256 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:04.256+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:04.324 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.191228672999998, 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:04.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:04.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:04.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.720115219999997, 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:04.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:04.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.489922841000002, 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.345 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.489215291999997, 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.351 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:04.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:04.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:04.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:04.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.488578298, 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:04.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:04.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:04.363 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.431513103, 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.364 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:04.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:04.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:04.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:04.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.430022332, 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.371 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:04.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.377 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.397672669999999, 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:04.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:04.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:04.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.391041895000001, 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.383 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:04.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:04.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:04.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.390391213999997, 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.388 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:04.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:04.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:04.393 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.386918522999999, 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:04.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:04.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.385779800999998, 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:04.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:04.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:04.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:04.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:04.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.384832103999997, 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.404 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.356468110000002, 2023-12-06T07:58:04.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.328258593000001, 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:04.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:04.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:04.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.718060659000002, 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.420 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.717415819000003, 2023-12-06T07:58:04.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:04.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:04.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.430 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.696949064999998, 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.435 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.696200148999999, 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.695558079999998, 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:04.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:04.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:04.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:04.446 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:04.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:04.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:04.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:04.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:04.449 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:04.449 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:04.448+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:04.513 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:04.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:04.548 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:04.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:04.617 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:04.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:04.621 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:04.621 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:04.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:04.671 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:04.671+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:04.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:04.702 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:04.750 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:04.749+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:04.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:04.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:04.806 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:04.815 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:04.815 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:04.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:04.866 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:04.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:04.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:04.896 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:04.951 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:04.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:04.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:04.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:05.006 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.006 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:05.053 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:05.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:05.154 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:05.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:05.185 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.185 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:05.215 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T07:58:05.277 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:05.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:05.344 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.344 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:05.359 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:05.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:05.459 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:05.459+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:05.479 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:05.523 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.523 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:05.577 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:05.576+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:05.661 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.661 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.665 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:05.664+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:05.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:05.767 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:05.766+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:05.849 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:05.849 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:05.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:05.902 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:05.900+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:05.971 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:05.970+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:06.027 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.028 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:06.071 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:06.071+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:06.175 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:06.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:06.214 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.214 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:06.279 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:06.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:06.359 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.359 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:06.378 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:06.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:06.479 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:06.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.498 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:06.593 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:06.590+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:06.674 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.675 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:06.684 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:06.683+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:06.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:06.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:06.784 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:06.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:06.856 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.856 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:06.885 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:06.885+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:06.986 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:06.986+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:06.994 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:06.994 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:06.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:07.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:07.087 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:07.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:07.142 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:07.143 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:07.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:07.192 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:07.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:07.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T07:58:07.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:07.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:07.295 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:07.294+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:07.320 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:07.320 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:07.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:07.331 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:07.332 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:07.414 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:07.413+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:07.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:07.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:07.496 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:07.496+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:07.500 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:07.500 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:07.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:07.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:07.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:07.598 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:07.597+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:07.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:07.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:07.660 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:07.660 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:07.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:07.698 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:07.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:07.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:07.799 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:07.799+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:07.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:07.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:07.839 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:07.840 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:07.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:07.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:07.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:07.906 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:07.906+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:08.000 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:08.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:08.027 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:08.027 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:08.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:08.027 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:08.027 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:08.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:08.102 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:08.102+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:08.167 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:08.168 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:08.168 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:08.168 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:08.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:08.202 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:08.202+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:08.304 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:08.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:08.345 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:08.345 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:08.345 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:08.345 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:08.345 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:08.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:08.398 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T07:58:08.405 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:08.405+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:08.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:08.505 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:08.505+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.418228223, 2023-12-06T07:58:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:08.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:08.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:08.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.947114769999999, 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:08.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:08.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:08.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.716922390999997, 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.564 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:08.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:08.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.716214841999999, 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:08.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:08.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:08.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:08.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.580 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.715577848000002, 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:08.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:08.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.658512653000003, 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.586 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:08.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.657021882000002, 2023-12-06T07:58:08.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:08.593 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.593 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.593 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:08.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:08.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.624672220000001, 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.602 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.618041445000003, 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.608 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:08.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.617390764, 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.614 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:08.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:08.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:08.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.613918073000001, 2023-12-06T07:58:08.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.612779351, 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.611831654, 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.583467659999997, 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.555258143000003, 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.945060208999998, 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.944415368999998, 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.923948615, 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.923199699000001, 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.92255763, 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:08.683 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:08.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:08.685 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:08.684+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:08.786 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:08.786+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:08.835 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:08.835 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:08.835 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:08.836 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:08.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:08.887 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:08.887+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:08.990 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:08.990+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:09.001 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:09.001 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:09.001 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:09.002 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:09.002 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:09.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:09.102 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:09.099+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:09.193 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:09.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:09.209 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:09.210 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:09.210 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:09.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:09.210 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:09.210 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:09.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:09.312 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:09.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:09.395 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:09.395+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:09.397 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:09.398 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:09.398 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:09.398 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:09.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:09.496 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:09.496+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:09.537 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.385 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:09.577 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:09.578 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:09.578 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:09.578 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:09.578 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:09.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:09.629 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:09.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:09.698 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:09.698+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.344576048, 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:09.766 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:09.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:09.782 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.630 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:09.799 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:09.799+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:09.900 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:09.899+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:09.902 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:09.902 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:09.902 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:09.902 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:09.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:09.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:10.001 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:10.001+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:10.090 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:10.091 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:10.091 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:10.091 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:10.091 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:10.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:10.102 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:10.101+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:10.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:10.186 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:10.203 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:10.203+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:10.229 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:10.229 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:10.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:10.229 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:10.229 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:10.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:10.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:10.299 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:10.325 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:10.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:10.366 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:10.366 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:10.366 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:10.366 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:10.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:10.384 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:10.384 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:10.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T07:58:10.420 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:10.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:10.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:10.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:10.505 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:10.505+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:10.509 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:10.509 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:10.509 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:10.509 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:10.509 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:10.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:10.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:10.582 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:10.607 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:10.607+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:10.683 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:10.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:10.707 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:10.707+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:10.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:10.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:10.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:10.808 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:10.808+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:10.857 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:10.858 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:10.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:10.887 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:10.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:10.912 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:10.911+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:10.998 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:10.999 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:11.010 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:11.010+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:11.110 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:11.110+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:11.178 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:11.178 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:11.212 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:11.211+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:11.313 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:11.313+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:11.357 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:11.357 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:11.414 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:11.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:11.493 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:11.493 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:11.545 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:11.543+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:11.617 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:11.616+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:11.645 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T07:58:11.673 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:11.673 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:11.724 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:11.723+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:11.732 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:11.732 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:11.733 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-12-06T07:58:11.733 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T07:58:11.733 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T07:58:11.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-12-06T07:58:11.819 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:11.819+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:11.834 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:11.834 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:11.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:11.875 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:11.875 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T07:58:11.875 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:11.928 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:11.927+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:12.012 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.013 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:12.025 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:12.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:12.126 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:12.126+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:12.197 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.198 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:12.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:12.227 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:12.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:12.329 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:12.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:12.382 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.382 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:12.430 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:12.429+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:12.520 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.521 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:12.532 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:12.531+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:12.632 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:12.632+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:12.726 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.726 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.733 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:12.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:12.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:12.834 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:12.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:12.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:12.911 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:12.911 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:12.918 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:12.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:12.938 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:12.934+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:13.036 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:13.036+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:13.104 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:13.104 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:13.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:13.155 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:13.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:13.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:13.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:13.238 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:13.238+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:13.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:13.265 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:13.322 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:13.322 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:13.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:13.339 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:13.338+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:13.388 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:13.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:13.389 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.237 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:13.440 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:13.439+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:13.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:13.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:13.494 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.342 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:13.531 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:13.532 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:13.532 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:13.532 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:13.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:13.580 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:13.579+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:13.580 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:13.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:13.642 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:13.642+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:13.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T07:58:13.681 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:13.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:13.702 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.550 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:13.710 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:13.710 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:13.711 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:13.711 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:13.711 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:13.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:13.763 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:13.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:13.780 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:13.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:13.844 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:13.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:13.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:13.867 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 34.481495371000001, 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:13.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:13.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:13.903 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:13.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:13.952 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:13.951+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:14.040 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.040 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:14.040 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:14.040 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:14.046 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:14.045+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:14.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:14.146 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:14.146+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:14.225 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.225 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:14.226 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:14.226 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:14.226 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:14.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:14.280 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:14.279+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:14.348 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:14.348+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:14.361 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.361 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:14.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:14.361 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:14.362 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:14.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:14.449 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:14.449+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:14.550 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:14.550+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:14.557 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.557 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:14.557 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:14.557 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:14.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:14.651 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:14.651+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:14.742 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:14.743 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:14.743 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:14.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:14.753 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:14.753+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:14.839 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T07:58:14.853 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:14.853+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.958788564000002, 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:14.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:14.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:14.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.295007562999999, 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:14.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:14.958 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.291726934000003, 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:14.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.962 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.986950317999998, 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:14.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.986107220999997, 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:14.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:14.972 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.963571715, 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:14.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.976 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.961579295999996, 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:14.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.978 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.978 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:14.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.961047031, 2023-12-06T07:58:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:14.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:14.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:14.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.944122122000003, 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.943285488000001, 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:14.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.997 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.918838219999998, 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:14.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.912935855999997, 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.908229859999999, 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.006 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.905927984999998, 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.010 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:15.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:15.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.014 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.905429662000003, 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.018 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.904975751000002, 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.903753350999999, 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:15.023 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:15.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.258587370000001, 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.032 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.251148954999998, 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:15.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:15.036 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.241075582999997, 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:15.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.040 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:15.042 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:15.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:15.143 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:15.143+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:15.174 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.174 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:15.174 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:15.174 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:15.246 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:15.246+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:15.317 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:15.317 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:15.365 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:15.364+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:15.448 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:15.447+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.493 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:15.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:15.493 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:15.548 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:15.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:15.632 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.632 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:15.632 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:15.633 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:15.684 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:15.683+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:15.749 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:15.749+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:15.767 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.767 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:15.767 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:15.767 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:15.768 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:15.850 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:15.850+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:15.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:15.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:15.949 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:15.949 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 35.372656419000002, 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:15.950 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:15.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:15.955 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:15.957 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:15.956+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:15.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:16.058 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:16.057+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:16.137 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:16.137 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:16.138 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:16.138 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:16.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:16.158 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:16.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:16.250 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:16.250 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:16.291 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:16.286+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:16.332 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:16.332 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:16.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:16.356 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:16.357 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:16.360 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:16.359+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:16.460 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:16.460+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:16.463 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:16.463 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:16.503 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:16.503 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:16.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:16.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:16.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:16.561 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:16.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:16.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:16.645 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:16.654 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:16.661 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:16.661+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:16.667 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T07:58:16.667 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T07:58:16.667 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:58:16.683 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:16.684 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:16.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:16.696 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T07:58:16.685+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86657) UID: 0 2023-12-06T07:58:16.698 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T07:58:16.687+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T07:58:16.735 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:16.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:16.762 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:16.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:16.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:16.854 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:16.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T07:58:16.856 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:16.856 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:16.863 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:16.862+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:16.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:16.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:16.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:16.885 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T07:58:16.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T07:58:16.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:16.967 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:16.967+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:17.065 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:17.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:17.073 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:17.074 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:17.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:17.167 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:17.167+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:17.238 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:08.318 86690== 2023-12-06T07:58:17.239 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:08.318 86690== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:58:17.249 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:58:17.260 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:17.260 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:17.266 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:17.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:17.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:17.368 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:17.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:17.458 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:17.458 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:17.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:17.470 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:17.470+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:17.583 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:17.583+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:17.613 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:17.614 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:17.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:17.710 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:17.710+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:17.773 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:17.773+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:17.796 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:17.796 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:17.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 29, 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:58:14.769619+0000", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T07:58:17.829 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T07:58:17.830 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:58:17.831 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:58:17.832 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:58:17.833 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:58:17.834 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:58:17.835 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T07:58:17.836 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:58:17.838 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T07:58:17.839 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:58:17.840 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:58:17.841 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:58:17.842 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 4.9800000190734863, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 4.9800000190734863, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 4.380000114440918, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 4.380000114440918, 2023-12-06T07:58:17.843 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.844 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.847 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.848 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.849 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 16, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T07:58:17.850 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.852 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.853 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T07:58:17.855 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T07:58:17.856 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.857 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.860 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.861 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:58:17.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.864 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:58:17.867 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T07:58:17.868 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:58:17.869 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 24, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:58:17.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:58:17.876 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T07:58:17.878 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.879 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:58:17.881 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:58:17.882 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:17.883 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:17.884 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:17.886 INFO:tasks.thrashosds.thrasher:j is {'epoch': 29, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T07:58:14.769619+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.6', 'osds': [7, 2]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T07:58:17.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:58:17.887 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:17.887+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:17.987 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:17.987+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:18.028 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.028 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:18.088 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:18.087+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:18.190 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:18.189+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:18.224 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.224 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:18.290 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:18.289+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:18.365 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.365 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:18.417 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:18.416+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:18.492 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:18.491+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:18.505 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:58:18.506 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:58:18.518 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2023-12-06T07:58:17.809790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218820,"kb_used_data":2656,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755900,"statfs":{"total":773094113280,"available":772870041600,"internally_reserved":0,"allocated":2719744,"data_stored":1522462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001349"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.996532+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:58:09.996532+0000","last_peered":"2023-12-06T07:58:09.996532+0000","last_clean":"2023-12-06T07:58:09.996532+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:58:09.996532+0000","last_undegraded":"2023-12-06T07:58:09.996532+0000","last_fullsized":"2023-12-06T07:58:09.996532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.763566+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T07:58:09.763566+0000","last_peered":"2023-12-06T07:58:09.763566+0000","last_clean":"2023-12-06T07:58:09.763566+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T07:58:09.763566+0000","last_undegraded":"2023-12-06T07:58:09.763566+0000","last_fullsized":"2023-12-06T07:58:09.763566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.990705+0000","last_change":"2023-12-06T07:57:30.956316+0000","last_active":"2023-12-06T07:58:09.990705+0000","last_peered":"2023-12-06T07:58:09.990705+0000","last_clean":"2023-12-06T07:58:09.990705+0000","last_became_active":"2023-12-06T07:57:30.951129+0000","last_became_peered":"2023-12-06T07:57:30.951129+0000","last_unstale":"2023-12-06T07:58:09.990705+0000","last_undegraded":"2023-12-06T07:58:09.990705+0000","last_fullsized":"2023-12-06T07:58:09.990705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:21:17.393399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-12-06T07:58:11.918687+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:58:11.918687+0000","last_peered":"2023-12-06T07:58:11.918687+0000","last_clean":"2023-12-06T07:58:11.918687+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:58:11.918687+0000","last_undegraded":"2023-12-06T07:58:11.918687+0000","last_fullsized":"2023-12-06T07:58:11.918687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.986950+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:58:09.986950+0000","last_peered":"2023-12-06T07:58:09.986950+0000","last_clean":"2023-12-06T07:58:09.986950+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:58:09.986950+0000","last_undegraded":"2023-12-06T07:58:09.986950+0000","last_fullsized":"2023-12-06T07:58:09.986950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.760879+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:58:09.760879+0000","last_peered":"2023-12-06T07:58:09.760879+0000","last_clean":"2023-12-06T07:58:09.760879+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:58:09.760879+0000","last_undegraded":"2023-12-06T07:58:09.760879+0000","last_fullsized":"2023-12-06T07:58:09.760879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.984020+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:58:09.984020+0000","last_peered":"2023-12-06T07:58:09.984020+0000","last_clean":"2023-12-06T07:58:09.984020+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:58:09.984020+0000","last_undegraded":"2023-12-06T07:58:09.984020+0000","last_fullsized":"2023-12-06T07:58:09.984020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:09.993793+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:58:09.993793+0000","last_peered":"2023-12-06T07:58:09.993793+0000","last_clean":"2023-12-06T07:58:09.993793+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:58:09.993793+0000","last_undegraded":"2023-12-06T07:58:09.993793+0000","last_fullsized":"2023-12-06T07:58:09.993793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":124,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-12-06T07:58:08.406824+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:58:08.406824+0000","last_peered":"2023-12-06T07:58:08.406824+0000","last_clean":"2023-12-06T07:58:08.406824+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:58:08.406824+0000","last_undegraded":"2023-12-06T07:58:08.406824+0000","last_fullsized":"2023-12-06T07:58:08.406824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476750,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":89768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.376999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.319000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.976000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.898}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.153}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.295000000000002}]}]},{"osd":2,"up_from":16,"seq":68719476751,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":89768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4969999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.468}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.789999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.548}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.015000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.749000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.323}]}]},{"osd":1,"up_from":16,"seq":68719476751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":487608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3470000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.963000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.449}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.912000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.539999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.506}]}]},{"osd":0,"up_from":16,"seq":68719476752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":89768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7759999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.182}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.156999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.433999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.898999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.417999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.657}]}]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":94087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8339999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.524000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1159999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.01}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.608000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.178999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.864000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":94087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0679999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4930000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8929999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.609}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.582000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.170999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542159,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":487608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.440000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.141999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.577999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.960000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.065999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.757999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":89768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.558}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.748000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.225999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.997999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.974}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.84}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.185}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:58:18.519 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [1, 6] 2023-12-06T07:58:18.519 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.5', '1', '6'] 2023-12-06T07:58:18.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.5 1 6 2023-12-06T07:58:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:18.596 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:18.596+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:18.693 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:18.693+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:18.725 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.725 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:18.794 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:18.794+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:18.865 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:18.865 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:18.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:18.919 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:18.918+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:18.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:19.002 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:19.045 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:19.047 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:19.012+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:19.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:19.100 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:19.099+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:19.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:19.202 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:19.203 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:19.202+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:19.217 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:19.217 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:19.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:19.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:19.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:19.315 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:19.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:19.355 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:19.355 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:19.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:19.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:19.375 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:19.414 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:19.413+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:19.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:19.458 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:19.494 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:19.494 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:19.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:19.504 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:19.503+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:19.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:19.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:19.604 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:19.603+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:19.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:19.642 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:19.690 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:19.690 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:19.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:19.744 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:19.743+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:19.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:19.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:19.807 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:19.807+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:19.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:19.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:19.871 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:19.871 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:19.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:19.908 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:19.908+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:20.011 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:20.011+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:20.048 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:20.049 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:20.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:20.112 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:20.111+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:20.190 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.614018223999999, 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:20.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:20.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:20.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:20.195 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:20.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:20.244 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:20.244+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:20.316 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:20.315+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:20.327 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:20.328 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:20.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:20.417 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:20.416+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:20.503 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:20.504 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:20.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:20.560 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:20.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:20.620 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:20.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:20.642 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:20.642 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:20.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:20.643 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:20.643 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:20.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:20.736 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:20.736+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:20.820 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:20.820 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:20.820 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:20.820 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:20.823 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:20.823+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:20.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:20.925 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:20.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:20.998 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:20.998 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:20.998 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:20.998 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:20.999 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:21.025 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:21.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:21.128 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:21.128+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.197 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 61.066181118999999, 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.595067665999998, 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:21.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:21.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:21.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.364875286999997, 2023-12-06T07:58:21.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:21.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.364167737999999, 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:21.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:21.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:21.220 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.363530744000002, 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:21.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:21.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:21.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.306465549000002, 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:21.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:21.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:21.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.304974778000002, 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:21.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.272625116, 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:21.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.265994341000003, 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.250 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.265343659999999, 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.255 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:21.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:21.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.261870969, 2023-12-06T07:58:21.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:21.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:21.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:21.265 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.260732247, 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:21.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.259784549999999, 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:21.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:21.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.231420556000003, 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:21.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:21.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.280 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.203211039000003, 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:21.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:21.285 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.593013104999997, 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:21.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.592368264999998, 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.291 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:21.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:21.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.571901511, 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:21.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:21.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.571152595000001, 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.301 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:21.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.570510526, 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:21.306 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:21.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:21.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:21.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:21.311 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T07:58:21.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:21.313 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:21.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:21.414 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:21.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:21.480 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:21.480 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:21.480 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:21.481 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:21.534 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:21.534+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:21.617 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:21.617 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:21.617 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:21.617 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:21.617 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.617 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:21.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:21.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:21.717 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:21.717+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:21.796 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:21.797 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:21.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:21.797 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:21.797 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:21.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:21.847 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:21.847+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:21.919 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:21.918+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:21.956 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:21.956 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:21.956 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:21.956 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:21.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:22.020 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:22.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:22.089 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:22.089 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:22.089 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:22.089 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:22.089 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:22.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:22.143 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:22.143+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:22.222 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:22.222+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:22.233 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:22.233 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:22.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.816384571, 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:22.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:22.237 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:22.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:22.323 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:22.323+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:22.381 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:22.381 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:22.381 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:22.381 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:22.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:22.425 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:22.425+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:22.518 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:22.519 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:22.519 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:22.519 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:22.519 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:22.526 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:22.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:22.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:22.627 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:22.627+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:22.696 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:22.696 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:22.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:22.696 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:22.697 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:22.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:22.749 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:22.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:58:22.771 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T07:58:22.772 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T07:58:22.773 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T07:58:22.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:22.829 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:22.829+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:22.876 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:22.876 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:22.876 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:22.876 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:22.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:22.931 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:22.930+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:23.031 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:23.031 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:23.031 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:23.031 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:23.031 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:23.031 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:23.030+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:23.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:23.132 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:23.131+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:23.232 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:23.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:23.265 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.284296885000003, 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:23.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.620515884, 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:23.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:23.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.617235254999997, 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:23.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:23.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:23.278 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.312458638999999, 2023-12-06T07:58:23.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:23.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:23.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:23.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.311615541999998, 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:23.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:23.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:23.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.289080036000001, 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:23.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:23.300 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.287087616999997, 2023-12-06T07:58:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.286555352000001, 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:23.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:23.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:23.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.269630442999997, 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:23.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:23.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:23.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:23.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.315 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.268793809000002, 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:23.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:23.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:23.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:23.320 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.244346540999999, 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:23.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.238444176999998, 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.233738181, 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.231436305999999, 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:23.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:23.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:23.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:23.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:23.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.340 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.230937982999997, 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:23.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.230484072000003, 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.345 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.229261672, 2023-12-06T07:58:23.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:23.353 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.584095691000002, 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.576657275999999, 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:23.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:23.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:23.361 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.566583903999998, 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:23.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:23.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:23.366 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:23.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:23.369 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:23.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:23.440 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:23.468 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:23.467+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:23.539 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:23.540 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:23.540 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:23.540 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:23.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:23.568 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:23.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:23.672 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:23.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:23.717 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:23.717 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:23.717 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:23.717 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:23.717 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:23.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:23.771 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:23.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:23.853 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:23.853 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:23.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:23.902 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:23.902+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:23.972 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:23.972+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:24.028 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:24.029 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:24.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:24.074 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:24.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:24.175 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:24.174+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:24.206 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:24.206 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:24.206 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:24.206 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:24.206 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:24.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:24.275 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:24.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:24.344 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:24.345 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 43.767820530000002, 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:24.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:24.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:24.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:24.352 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:24.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:24.376 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:24.376+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:24.477 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:24.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:24.486 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:24.486 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:24.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:24.578 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:24.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:24.626 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:24.626 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:24.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:24.678 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:24.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:24.779 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:24.779+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:24.804 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:24.804 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:24.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:24.881 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:24.880+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:24.982 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:24.981+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:24.985 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:24.985 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:24.985 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:24.986 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:24.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:25.083 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:25.083+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:25.167 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:25.167 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:25.167 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:25.167 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:25.167 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:25.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:25.225 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:25.224+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:25.285 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:25.284+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:25.386 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:25.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:25.400 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:25.400 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:25.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 65.269192606000004, 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.798079153000003, 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:25.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:25.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:25.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.567886774000002, 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:25.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:25.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:25.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.567179224999997, 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.418 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:25.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.566542231, 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:25.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:25.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:25.428 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.509477036, 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.429 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:25.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:25.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:25.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.507986265, 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:25.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.475636602999998, 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:25.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:25.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.469005828, 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.468355146999997, 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.464882455999998, 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:25.457 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.463743733999998, 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:25.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:25.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:25.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.462796036999997, 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.469 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.434432043000001, 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:25.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:25.480 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.406222526000001, 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.481 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:25.486 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.796024592000002, 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:25.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.795379752000002, 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.774912997999998, 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:25.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:25.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.774164081999999, 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:25.503 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:25.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:25.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:25.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.773522012999997, 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:25.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:25.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:25.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:25.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:25.514 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:25.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:25.516 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:25.515+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:25.617 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:25.616+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:25.684 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:25.684 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:25.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:25.739 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:25.739+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:25.818 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:25.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:25.823 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:25.823 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:25.823 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:25.823 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:25.823 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:25.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:25.919 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:25.919+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:26.001 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:26.001 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:26.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:26.001 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:26.001 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:26.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:26.019 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:26.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:26.121 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:26.120+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:26.143 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:26.143 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:26.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:26.222 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:26.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:26.284 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:26.284 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:26.284 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:26.284 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:26.284 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:26.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:26.324 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:26.323+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:26.423 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:26.423+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:26.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:26.466 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:26.469 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:26.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:26.469 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:26.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.049224504999998, 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:26.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:26.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:26.473 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:26.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:26.524 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:26.524+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:26.614 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:26.614 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:26.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:26.626 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:26.625+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:26.727 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:26.726+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:26.750 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:26.750 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:26.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:26.828 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:26.828+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:26.927 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:26.927 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:26.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:26.927 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:26.927 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:26.929 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:26.929+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:26.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:27.030 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:27.029+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:27.066 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:27.066 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:27.067 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:27.067 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:27.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:27.133 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:27.132+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:27.233 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:27.233+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:27.244 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:27.244 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:27.244 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:27.244 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:27.244 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:27.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:27.334 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:27.333+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:27.423 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.442727933, 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:27.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.778946931999997, 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:27.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:27.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.431 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.775666302999994, 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.470889686999996, 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:27.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:27.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:27.439 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.470046589999995, 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.447511083999999, 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:27.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:27.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:27.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.445518664999994, 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.444986400000005, 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:27.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.454 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.428061490999994, 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.427224856999999, 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:27.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:27.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:27.462 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.402777588999996, 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.396875225000002, 2023-12-06T07:58:27.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:27.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:27.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.392169229000004, 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.470 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.389867354000003, 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:27.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:27.477 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.389369031000001, 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:27.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.481 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.388915119999993, 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:27.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.387692720000004, 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:27.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:27.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.742526738999999, 2023-12-06T07:58:27.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:27.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:27.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.493 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.735088324000003, 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.725014952000002, 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:27.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:27.501 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:27.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:27.503 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:27.502+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:27.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T07:58:27.603 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:27.603+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:27.671 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:27.672 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:27.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:27.704 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:27.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:27.808 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:27.805+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:27.849 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:27.849 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:27.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:27.947 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:27.945+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:28.008 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:28.008+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:28.030 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:28.030 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:28.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:28.031 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:28.031 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:28.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:28.124 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:28.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:28.208 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:28.208 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:28.208 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:28.208 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:28.209 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:28.210+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:28.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:28.311 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:28.311+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:28.384 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:28.384 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:28.384 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:28.384 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:28.384 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:28.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:28.412 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:28.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:28.514 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:28.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.563 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.986804821, 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:28.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:28.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:28.567 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:28.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:28.618 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:28.618+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:28.713 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:28.714 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:28.714 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:28.714 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:28.719 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:28.715+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:28.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:28.816 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:28.816+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:28.890 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:28.890 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:28.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:28.923 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:28.918+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:29.019 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:29.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:29.027 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:29.027 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:29.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:29.027 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:29.027 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:29.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:29.120 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:29.120+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:29.212 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:29.212 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:29.212 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:29.212 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:29.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:29.222 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:29.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:29.322 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:29.322+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:29.349 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:29.350 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:29.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:29.445 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:29.445+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:29.525 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:29.524+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:29.583 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:29.583 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:29.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 69.45316579, 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:29.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.591 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:29.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.982052336999999, 2023-12-06T07:58:29.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:29.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.596 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.751859957999997, 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:29.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.608 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.751152408999999, 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.750515414999995, 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.693450220000003, 2023-12-06T07:58:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.691959448999995, 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.659609786999994, 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.652979012000003, 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.652328331000007, 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.648855639999994, 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.647716918, 2023-12-06T07:58:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.646769221, 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.618405226999997, 2023-12-06T07:58:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.590195710000003, 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.979997775999998, 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.979352935999998, 2023-12-06T07:58:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.958886182000001, 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.958137266000001, 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.957495197, 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:29.703 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:29.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:29.705 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:29.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:29.806 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:29.805+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:29.880 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:29.880 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:29.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:29.933 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:29.933+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:30.008 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:30.007+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:30.019 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:30.019 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:30.019 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:30.019 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:30.019 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:30.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:30.109 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:30.108+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:30.198 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:30.199 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:30.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:30.199 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:30.199 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:30.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:30.210 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:30.210+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:30.311 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:30.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:30.338 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:30.339 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:30.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:30.432 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:30.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:30.513 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:30.512+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:30.515 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:30.515 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:30.515 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:30.515 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:30.515 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:30.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:30.613 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:30.613+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.697 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 51.280330712999998, 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:30.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:30.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:30.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:30.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:30.702 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:30.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:30.752 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:30.751+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:30.816 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:30.816+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:30.848 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:30.848 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:30.848 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:30.848 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:30.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:30.918 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:30.917+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:31.019 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:31.018+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:31.027 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:31.028 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:31.119 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:31.120+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:31.204 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:31.204 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:31.220 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:31.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:31.325 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:31.321+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:31.350 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:31.351 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:31.422 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:31.422+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:31.492 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:31.492 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:31.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:31.539 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:31.539 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:31.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:31.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:31.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:31.624 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:31.624+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:31.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:31.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:31.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 70.732627276000002, 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 70.068846274999999, 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 70.065565645999996, 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.760789029999998, 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.759945932999997, 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.737410427, 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.735418007999996, 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.734885743000007, 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.717960833999996, 2023-12-06T07:58:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.717124200000001, 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.692676931999998, 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.686774568000004, 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.682068572000006, 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.679766697000005, 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.679268374000003, 2023-12-06T07:58:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.678814462999995, 2023-12-06T07:58:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:31.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 68.677592063000006, 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:31.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:31.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:31.796 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 63.032426082000001, 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:31.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 63.024987666999998, 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:31.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.804 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 63.014914294999997, 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:31.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:31.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:31.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:31.808 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:31.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:31.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:31.823 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:31.823+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:31.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:31.883 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:31.912 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:31.912+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:31.941 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:31.941 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:31.941 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:31.941 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:31.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:31.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:31.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:32.013 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:32.013+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:32.066 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:32.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:32.120 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.120 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:32.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:32.154 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:32.155 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:32.153+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:32.219 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:32.219+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:32.286 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.287 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:32.338 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:32.337+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:32.381 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.5 pg_upmap mapping to [1,6] 2023-12-06T07:58:32.381 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T07:58:32.421 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:32.421+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:32.504 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.505 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:32.556 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:32.555+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:32.625 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:32.624+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:32.686 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:32.738 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:32.737+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:32.823 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.823 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:32.833 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:32.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:32.928 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:32.928+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:32.963 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:32.963 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:32.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:33.057 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:33.057+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:33.102 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:33.115 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T07:58:33.115 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T07:58:33.115 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T07:58:33.145 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:33.145 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:33.145 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:33.145+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:33.147 INFO:tasks.ceph.mon.c:Started 2023-12-06T07:58:33.147 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T07:58:33.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:33.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:33.239 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:33.239+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:33.342 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:33.341+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:33.356 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:33.357 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:33.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:33.432 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:33.431+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:33.533 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:33.533+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:33.544 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:33.545 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:33.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:33.637 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:33.634+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:33.736 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:33.736+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:33.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:33.837 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:33.836+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:33.916 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:33.917 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:33.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:33.937 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:33.937+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:34.029 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:34.038 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:34.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:34.060 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:34.061 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:34.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:34.139 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:34.138+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:34.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:34.245 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:34.245 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:34.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:34.329 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:34.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:34.341 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:34.340+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:34.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T07:58:34.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:34.440 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:34.441 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:34.441+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:34.462 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:34.462 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:34.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:34.543 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:34.542+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:34.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:34.564 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:34.645 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:34.644+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:34.653 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:34.653 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:34.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:34.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:34.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:34.746 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:34.746+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:34.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:34.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:34.800 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:34.800 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:34.800 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:34.800 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:34.800 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:34.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:34.844 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:34.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:34.848 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:34.847+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:34.942 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:34.942 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:34.948 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:34.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:35.012 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 55.594876479, 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:35.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:35.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:35.016 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:35.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:35.031 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:35.031 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:35.048 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:35.048+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:35.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:35.119 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:35.149 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:35.149+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:35.153 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:35.153 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:35.153 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:35.153 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:35.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:35.252 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:35.251+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:35.330 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:35.330 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:35.330 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:35.330 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:35.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:35.353 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:35.353+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:35.454 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:35.454+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:35.514 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:35.514 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:35.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:35.514 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:35.514 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:35.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:35.565 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:35.565+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:35.657 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:35.657+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:35.694 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:35.694 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:35.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:35.758 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:35.758+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:35.834 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:35.835 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:35.835 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:35.835 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:35.835 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:35.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:35.890 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:35.889+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:35.960 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:35.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:35.992 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T07:58:36.025 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.025 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:36.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:36.025 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 75.038407968000001, 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:36.026 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:36.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:36.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:36.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:36.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.035 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.374626966999998, 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:36.036 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:36.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:36.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:36.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:36.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:36.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:36.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.044 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.371346337999995, 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:36.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:36.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.066569721999997, 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.065726624999996, 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:36.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:36.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.043191118999999, 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:36.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.041198699999995, 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:36.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:36.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.040666435000006, 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:36.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:36.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:36.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.072 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.023741525999995, 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.022904892, 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:36.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.998457623999997, 2023-12-06T07:58:36.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:36.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.992555260000003, 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:36.085 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.086 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.086 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:36.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.987849264000005, 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:36.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.099 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.985547389000004, 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:36.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:36.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:36.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.985049066000002, 2023-12-06T07:58:36.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:36.107 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.107 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.107 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:36.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:36.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:36.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.113 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.984595154999994, 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.114 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:36.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:36.118 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 72.983372755000005, 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.338206774, 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:36.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.126 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.330768359000004, 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.320694986999996, 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:36.135 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:36.137 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.136+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.237 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:36.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:36.303 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:36.304 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:36.304 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:36.359 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.438 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.438+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.442 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.442 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:36.442 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:36.442 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:36.442 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:36.539 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.538+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.623 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:36.624 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:36.670 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.741 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:36.740+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:36.797 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.798 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:36.798 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:36.798 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:36.842 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:36.842+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:36.943 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:36.943+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:36.979 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:36.979 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:36.979 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:36.979 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:36.979 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:36.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:37.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:37.045 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:37.044+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:37.118 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:37.118 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 56.541816466999997, 2023-12-06T07:58:37.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:37.136 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:37.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:37.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:37.182 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:37.182+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:37.248 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:37.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:37.275 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:37.276 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:37.276 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:37.276 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:37.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:37.350 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:37.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:37.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:37.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:37.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:37.422 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:37.471 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:37.470+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:37.485 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:37.485 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:37.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:37.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:37.574 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:37.574 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:37.574+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:37.652 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:37.651+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:37.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:37.680 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:37.685 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:37.686 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:37.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:37.782 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:37.782+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:37.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:37.790 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:37.803 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:37.862 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:37.858+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:37.870 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:37.870 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:37.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:37.879 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:37.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:37.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:37.964 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:37.964 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:37.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:38.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T07:58:38.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:38.065 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:38.066 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:38.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:38.110 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.110 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:38.153 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:38.154 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:38.157 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:38.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:38.259 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.259 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.259 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:38.258+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:38.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:38.359 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:38.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:38.439 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.439 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:38.460 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:38.460+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:38.561 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:38.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:38.587 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:29.635 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.590 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:29.638 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.610 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:29.658 86163== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.631 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.631 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:38.647 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:37.494 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.650 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:37.498 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.662 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:38.662+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:38.670 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:37.518 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:38.763 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:38.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:38.793 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.794 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:38.864 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:38.864+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:38.965 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:38.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:38.974 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:38.975 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:38.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:39.068 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:39.067+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:39.168 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:39.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:39.268 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:39.267+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:39.387 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:39.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:39.470 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:39.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:39.472 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.473 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:39.564 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T07:58:39.570 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:39.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:39.633 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.634 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:39.687 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:39.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:39.769 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.770 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.771 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:39.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:39.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:39.872 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:39.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:39.947 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:39.948 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:39.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:40.004 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:40.001+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:40.075 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:40.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:40.086 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.086 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:40.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:40.213 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:40.212+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:40.269 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.269 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.275 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:40.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:40.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:40.377 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:40.375+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:40.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:40.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:40.449 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:40.477 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:40.476+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:40.497 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:40.497 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:40.580 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:40.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:40.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:40.585 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:40.590 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.590 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:40.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:40.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:40.680 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:40.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:40.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:40.767 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:40.772 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.772 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.780 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:40.780+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:40.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:40.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:40.854 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:40.854 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:40.882 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:40.881+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:40.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:40.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:40.971 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:40.971 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:40.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:41.021 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:41.021+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:41.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:41.084 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:41.084+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:41.148 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:41.149 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:41.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:41.185 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:41.183+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:41.285 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:41.285+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:41.330 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:41.330 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:41.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.753676425999998, 2023-12-06T07:58:41.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:41.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:41.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:41.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:41.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:41.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:41.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:41.339 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:41.339 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:41.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:41.386 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:41.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:41.421 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.211 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:41.467 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:41.479 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:41.479 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:41.479 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:41.479 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:41.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:41.490 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:41.486+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:41.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T07:58:41.645 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:41.643+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:41.659 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:41.659 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:41.659 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:41.660 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:41.660 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:41.665 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.455 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:41.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:41.710 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:41.709+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:41.790 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:41.789+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:41.846 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:41.846 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:41.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:41.892 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:41.891+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:41.992 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:41.991+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:42.052 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:42.053 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:42.053 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:42.093 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:42.093+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:42.192 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.192 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:42.192 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:42.192 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:42.192 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.194 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:42.194+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:42.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:42.294 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:42.294+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:42.396 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:42.395+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:42.401 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:42.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:42.401 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-12-06T07:58:42.401 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T07:58:42.401 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T07:58:42.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-12-06T07:58:42.435 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 82.279108676000007, 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.807995223000006, 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.577802844000004, 2023-12-06T07:58:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.577095295000007, 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.576458301000002, 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.519393105999995, 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.517902335000002, 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.485552673000001, 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.478921897999996, 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.478271217, 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.474798526000001, 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.473659803999993, 2023-12-06T07:58:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.472712107000007, 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.444348113000004, 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.416138595999996, 2023-12-06T07:58:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.805940661999998, 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.805295822000005, 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.784829067999993, 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.784080152000001, 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.783438082999993, 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:42.561 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:42.564 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.564 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T07:58:42.564 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.564 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:42.561+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:42.664 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:42.663+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:42.731 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.731 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:42.731 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:42.731 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:42.785 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:42.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:42.866 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:42.865+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:42.869 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:42.869 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:42.869 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:42.869 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:42.869 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:42.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:42.967 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:42.967+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:43.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:43.047 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:43.047 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:43.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:43.047 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:43.047 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:43.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:43.068 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:43.067+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:43.168 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:43.168+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:43.244 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:43.244 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:43.244 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:43.244 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:43.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:43.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:43.293 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:43.294 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:43.293+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:43.371 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:43.369+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:43.399 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:43.400 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:43.400 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:43.400 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:43.400 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:43.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:43.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:43.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:43.490 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:43.491 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:43.491 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:43.490+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:43.573 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:43.572+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:43.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:43.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:43.625 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:43.625 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:43.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:43.681 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:43.680+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:43.681 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:43.682 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:43.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:43.768 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:43.768 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:43.768 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:43.775 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:43.775+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:43.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:43.855 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:43.855 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:43.876 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:43.876+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:43.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:43.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:43.977 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:43.976+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:44.078 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:44.077+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:44.139 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.139 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:44.179 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:44.178+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:44.280 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:44.279+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:44.283 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.284 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:44.380 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:44.380+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:44.465 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.465 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:44.481 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:44.480+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:44.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:44.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T07:58:44.608 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:44.607+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:44.647 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.647 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:44.683 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:44.682+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:44.784 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:44.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:44.785 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.785 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:44.885 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:44.884+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:44.966 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:44.966 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:44.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:45.018 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:45.017+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:45.086 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:45.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:45.145 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.145 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:45.175 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:45.195 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:45.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:45.289 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:45.288+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:45.290 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.080 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:45.322 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.322 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:45.343 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.133 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:45.417 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:45.416+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:45.491 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:45.490+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:45.503 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.503 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:45.569 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.358 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:58:45.598 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:45.597+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:45.679 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.680 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:45.693 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:45.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:45.794 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:45.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:45.829 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.829 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:45.895 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:45.894+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:45.969 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:45.969 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:45.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:45.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:46.019 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:46.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:46.097 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:46.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:46.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:46.153 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:46.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:46.199 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:46.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:46.299 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:46.298+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:46.303 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:46.304 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:46.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:46.400 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:46.399+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:46.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:46.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:46.501 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:46.500+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:46.526 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:46.526 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:46.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:46.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:46.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:46.601 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:46.601+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:46.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:46.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:46.691 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T07:58:46.703 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:46.702+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:46.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:46.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.654874746000004, 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.787 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.183761293000003, 2023-12-06T07:58:46.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:46.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:46.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.953568914000002, 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.799 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:46.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:46.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:46.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.952861365000004, 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:46.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:46.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:46.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.809 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.952224371, 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.810 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.895159175999993, 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.893668405, 2023-12-06T07:58:46.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:46.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:46.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:46.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.825 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.861318742999998, 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.854687967999993, 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.831 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.854037286999997, 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.850564595999998, 2023-12-06T07:58:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:46.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.849425874000005, 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.848478177000004, 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:46.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:46.857 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.820114183000001, 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:46.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:46.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:46.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.791904665999994, 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:46.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:46.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.181706731999995, 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:46.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:46.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.181061892000002, 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:46.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.160595138000005, 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:46.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:46.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:46.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:46.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.883 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.159846221999999, 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.159204153000005, 2023-12-06T07:58:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:46.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.894 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:46.895 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:46.895 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:46.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:46.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:46.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:46.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:46.940 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:46.940 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:46.939+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:46.997 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:46.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:47.027 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.027 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:47.027 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:47.027 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:47.033 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:47.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:47.121 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:47.121+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:47.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:47.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:47.198 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:47.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:47.208 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.209 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:47.209 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:47.209 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:47.209 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:47.300 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:47.299+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:47.351 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.351 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:47.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:47.351 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:47.351 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:47.400 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:47.400+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:47.489 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:47.489 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:47.501 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:47.501+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:47.565 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:47.565 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:47.565 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-12-06T07:58:47.565 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2023-12-06T07:58:47.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2023-12-06T07:58:47.602 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:47.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.650 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:47.651 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:47.651 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:47.651 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:47.702 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:47.702+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:47.801 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:47.801 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 68.384569299000006, 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.802 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:47.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:47.805 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:47.807 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:47.807+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:47.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:47.907 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:47.907+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:47.940 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:47.940 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:47.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:48.008 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:48.008+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:48.078 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:48.078 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:48.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:48.131 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:48.130+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:48.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:48.213 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:48.214 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:48.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:48.237 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:48.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:48.310 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:48.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:48.392 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:48.393 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:48.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:48.411 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:48.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:48.512 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:48.512+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:48.572 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:48.572 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:48.572 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:48.572 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:48.572 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:48.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:48.623 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:48.622+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:48.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T07:58:48.710 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.6 primary-affinity to 1 (8655362) 2023-12-06T07:58:48.714 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:48.714+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:48.802 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:48.802 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 87.818376690999997, 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:48.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:48.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:48.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 87.154595689999994, 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.807 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:48.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:48.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:48.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 87.151315061000005, 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:48.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:48.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:48.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.846538444999993, 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:48.816 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:48.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.845695348000007, 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:48.824 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.823159841999995, 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.828 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.821167423000006, 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.820635158000002, 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:48.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.803710249000005, 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:48.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:48.841 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.802873614999996, 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:48.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.778426347000007, 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:48.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:48.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:48.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.772523982999999, 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:48.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.767817987000001, 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:48.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:48.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:48.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.765516112, 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.765017788999998, 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:48.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.764563878000004, 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.868 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 85.763341478000001, 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:48.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:48.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 80.118175496999996, 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:48.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:48.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 80.110737082, 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.882 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:48.886 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 80.100663710000006, 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:48.891 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:48.893 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:48.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:48.895 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T07:58:48.895 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T07:58:48.895 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:58:48.895 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:48.893+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:48.995 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:48.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:49.065 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:49.065 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:49.065 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:49.065 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:49.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:49.096 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:49.096+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:49.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:49.172 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T07:58:48.910+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86159) UID: 0 2023-12-06T07:58:49.173 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T07:58:49.160+0000 9f54640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2023-12-06T07:58:49.197 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:49.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:49.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:49.297 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:49.297+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:49.398 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:49.397+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:49.433 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:49.433 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:49.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:49.433 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:49.433 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:49.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:49.500 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:49.500+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:49.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:49.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:49.601 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:49.601+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:49.655 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:49.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:49.655 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:49.655 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:49.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:49.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:49.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:49.706 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:49.705+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:49.787 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:49.788 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:49.803 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:49.803+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:49.834 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:49.834 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:49.834 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:49.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:49.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:49.879 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:49.904 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:49.904+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:49.912 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:40.960 86163== 2023-12-06T07:58:49.912 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:40.960 86163== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:58:49.923 INFO:tasks.ceph.mon.b.smithi107.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:58:49.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:49.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:50.002 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:50.002 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:50.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 69.426096954000002, 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:50.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:50.006 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:50.007 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:50.007 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:50.007 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:50.007 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:50.008 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:50.008+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:50.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:50.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:50.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:50.123 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:50.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:50.141 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.142 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:50.180 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:50.210 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:50.210+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:50.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:50.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:50.281 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.281 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:50.311 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:50.311+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:50.411 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:50.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:50.457 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.457 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:50.512 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:50.512+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:50.593 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.593 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:50.613 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:50.613+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:50.717 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:50.717+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:50.731 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.731 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:50.827 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:50.825+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:50.909 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:50.909 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:50.917 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:50.917+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:50.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:51.018 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:51.018+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:51.089 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.089 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:51.137 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:51.137+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:51.220 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:51.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:51.264 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.264 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:51.321 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:51.321+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:51.422 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:51.422+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:51.441 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.441 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:51.523 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:51.522+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:51.580 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.581 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:51.623 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:51.624+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:51.725 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:51.724+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:51.759 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.760 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:51.853 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:51.853+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:51.927 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:51.927+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:51.935 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:51.935 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:51.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:52.028 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:52.028+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:52.085 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:52.085 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:52.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:52.129 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:52.129+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:52.229 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:52.230+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:52.262 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:52.262 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:52.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:52.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:58:52.360 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:52.361+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:52.432 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:52.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:52.448 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:52.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:52.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:52.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:58:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:58:52.543 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:52.543+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:52.612 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:58:52.612 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:58:52.626 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:52.626 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:52.633 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:52.634+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:52.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:52.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T07:58:52.698 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:58:52.698 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:58:52.734 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:52.735+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:52.780 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:58:52.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:58:52.805 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:58:52.805 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:58:52.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:52.857 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:52.857+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:52.868 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:58:52.868 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:58:52.936 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:52.936+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:52.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:58:52.955 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:58:53.037 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 92.055036720000004, 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:53.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:53.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:53.041 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.391255719, 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.387975089999998, 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:53.046 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.083198474, 2023-12-06T07:58:53.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:53.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:53.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.082355376999999, 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:53.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:53.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.059819871000002, 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:53.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.057827451999998, 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.066 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:53.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:53.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:53.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.070 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.057295186999994, 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:53.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:53.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.040370277999997, 2023-12-06T07:58:53.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:53.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:53.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:53.081 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 91.039533644000002, 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:53.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:53.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:53.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.015086375999999, 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.086 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.009184012000006, 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:53.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:53.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:53.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.004478015999993, 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:53.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:53.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.002176141000007, 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.108 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:53.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.001677818000005, 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:53.117 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.001223906999996, 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:53.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:53.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:53.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:53.129 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.129 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 90.000001506999993, 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:53.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:53.135 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 84.354835526000002, 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:53.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:53.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 84.347397111000006, 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:53.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:53.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 84.337323738999999, 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:53.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:53.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:53.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:53.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:53.153 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:53.153 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.153 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:53.037+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:53.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:53.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:58:53.156 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:58:53.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:58:53.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:58:53.196 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:53.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:53.257 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:53.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:53.324 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.325 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:53.325 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:53.325 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:53.358 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:53.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:53.459 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:53.459+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:53.502 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.503 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:53.503 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:53.503 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:53.503 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:53.560 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:53.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:53.642 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:53.643 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:53.696 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:53.696+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:53.727 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:58:53.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:58:53.727 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-12-06T07:58:53.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T07:58:53.762 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:53.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:53.779 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.779 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:53.779 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:53.780 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:53.863 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:53.863+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:53.956 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:53.956 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:53.957 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:53.957 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:53.957 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:53.964 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:53.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:53.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:54.065 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:54.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:54.095 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:54.095 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:54.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.518178039999995, 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.096 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:54.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:54.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:54.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:54.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:54.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:54.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:54.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:54.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:54.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:54.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:54.103 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:54.103 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:54.103 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:54.103 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:54.103 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:54.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:54.166 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:54.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:54.235 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:54.235 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:54.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:54.267 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:54.268+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:54.369 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:54.369+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:54.413 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:54.413 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:54.413 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:54.413 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:54.413 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:54.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:54.470 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:54.471+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:54.571 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:54.571+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:54.590 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:54.590 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:54.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:54.590 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:54.591 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:54.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:54.672 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:54.672+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:54.769 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:54.770 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:54.772 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:54.772+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:54.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:54.873 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:54.873+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:54.944 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:54.944 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:54.944 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:54.944 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:54.944 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:54.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:54.994 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:54.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:58:54.997 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T07:58:54.997 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T07:58:54.997 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T07:58:54.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:55.076 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:55.076+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:55.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.048358811, 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.577245357999999, 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.347052978999997, 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.34634543, 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.345708435999995, 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.288643241000003, 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.287152469999995, 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.254802807999994, 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.248172033000003, 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.247521352000007, 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.244048660999994, 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.242909939, 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.241962242, 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.213598247999997, 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.185388731000003, 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.575190797000005, 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.574545956999998, 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.554079203000001, 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.553330286999994, 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.552688218, 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:55.299 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:55.299 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:55.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:55.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:55.302 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:55.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:55.403 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:55.403+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:55.479 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:55.480 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:55.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:55.532 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:55.532+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:55.604 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:55.604+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:55.618 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:55.618 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:55.618 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:55.619 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:55.619 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:55.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:55.699 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:58:55.705 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:55.705+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:55.798 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:55.798 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:55.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:55.798 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:55.798 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:55.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:58:55.807 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:55.807+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:55.907 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:55.907+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:55.933 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:55.934 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:55.934 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:55.934 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:55.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:58:56.009 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:56.009+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:56.109 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:56.109+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:56.111 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:56.111 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:56.111 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:56.112 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:56.112 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:56.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:58:56.211 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:56.211+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:56.292 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 76.875450681000004, 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:56.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:56.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:56.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:56.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:56.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:56.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:56.297 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:56.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:58:56.348 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:56.348+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:56.430 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:56.430+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:56.444 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:56.444 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:56.444 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:56.444 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:56.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:58:56.539 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:56.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:56.614 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:56.613+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:56.622 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:56.622 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:56.622 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:56.622 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:56.623 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:56.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:58:56.715 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:56.715+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:56.801 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:56.801 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:56.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:56.801 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:56.801 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:56.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:58:56.852 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:56.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:56.921 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:56.922+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:56.936 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:56.936 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:56.936 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:56.936 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:56.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:58:57.030 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:57.031+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:57.114 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:57.114 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:57.114 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:57.114 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:57.114 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:57.118 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:57.119+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:57.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:58:57.219 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:57.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:57.320 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:57.320+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.350909368999993, 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:58:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:58:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.687128368000003, 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.683847739000001, 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:58:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.379071123000003, 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:58:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.378228026000002, 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:58:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.355692520000005, 2023-12-06T07:58:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:58:57.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.353700101000001, 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:58:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:58:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.353167835999997, 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:57.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:58:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:58:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.336242927000001, 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.335406293000005, 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:58:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:58:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:58:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.310959025000002, 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:58:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:58:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:58:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.305056660999995, 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:58:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.300350664999996, 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:58:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:58:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.298048789999996, 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:58:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:58:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:58:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:58:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.297550466999994, 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:58:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:58:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.297096556, 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:58:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:58:57.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:58:57.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.295874155999996, 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:58:57.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:58:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 88.650708175000005, 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 88.643269759999995, 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:58:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:58:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 88.633196388000002, 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:58:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:57.421 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:57.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:58:57.423 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:57.422+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:57.523 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:57.524+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:57.589 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:57.590 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:57.590 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:57.590 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:57.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:58:57.624 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:57.624+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:57.726 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:57.726+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:57.767 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:57.768 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:57.768 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:57.768 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:57.768 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:57.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:58:57.827 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:57.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:57.929 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:57.929+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:57.943 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:57.944 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:57.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:57.944 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:58:57.944 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:57.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:58:58.029 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:58.029+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:58.120 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:58.121 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:58.121 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:58:58.121 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:58.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:58:58.131 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:58.131+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:58.233 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:58.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:58.297 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:58.298 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:58:58.298 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:58:58.298 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:58:58.298 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:58.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:58:58.348 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:58.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:58.435 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:58.435+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:58.477 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 77.901025121999993, 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:58:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:58:58.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:58.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:58:58.484 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:58:58.484 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:58:58.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:58:58.536 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:58:58.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:58:58.616 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:58.616 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:58.616 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:58.616 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:58.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:58:58.639 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:58.639+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:58.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:58:58.739 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:58.740+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:58.800 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:58.800 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:58.800 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:58.800 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:58.800 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:58.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:58:58.840 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:58.841+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:58.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T07:58:58.939 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:58:58.939 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:58.942 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:58.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:58.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:58:59.043 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:59.043+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:59.113 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:59.113 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:59.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:58:59.146 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:58:59.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:58:59.245 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:58:59.246+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:58:59.252 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:59.253 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:59.253 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:59.253 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:59.253 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:59.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:58:59.346 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:59.347+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:59.447 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:59.447 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.316600421999993, 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:58:59.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:58:59.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:58:59.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:58:59.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:58:59.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.845486969000007, 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.615294590000005, 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.462 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:58:59.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:58:59.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:58:59.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.614587040999993, 2023-12-06T07:58:59.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:59.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:58:59.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:58:59.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:58:59.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:58:59.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.613950047000003, 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:59.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:58:59.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.556884851999996, 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:58:59.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:58:59.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.555394081000003, 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:58:59.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:58:59.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:58:59.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.523044419000001, 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:58:59.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:58:59.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.516413643999996, 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:58:59.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:58:59.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:58:59.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.515762963, 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:58:59.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:58:59.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.512290272000001, 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.511 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:58:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.511151549999994, 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:58:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:58:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:58:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:58:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.510203852999993, 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:58:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:58:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.481839859000004, 2023-12-06T07:58:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:58:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:58:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:58:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.453630341999997, 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:58:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.843432407999998, 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:58:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:58:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:58:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.842787568000006, 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:58:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:58:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:58:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.822320813999994, 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:58:59.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.821571898000002, 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.820929828999994, 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:58:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:58:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:58:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:58:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:58:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:58:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:58:59.564 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:59.564 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:58:59.446+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:58:59.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:58:59.606 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:58:59.606+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:58:59.669 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:58:59.667+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:58:59.736 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:59.736 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:59.737 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:58:59.737 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:59.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:58:59.768 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:58:59.769+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:58:59.870 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:59.870+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:58:59.915 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:58:59.915 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:58:59.915 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:58:59.915 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:58:59.915 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:58:59.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:58:59.970 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:58:59.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:00.053 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:00.053 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:00.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:00.053 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:00.053 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:00.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:00.071 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:00.072+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:00.173 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:00.172+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:00.196 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:00.197 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:00.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:00.290 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:00.290+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:00.372 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:00.372 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:00.372 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:00.372 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:00.372 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:00.373 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:00.373+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:00.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:00.474 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:00.474+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:00.550 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:00.550 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:00.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.133545253999998, 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:00.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:00.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:00.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:00.555 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:00.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:00.605 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:00.605+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:00.676 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:00.676+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:00.702 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:00.703 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:00.703 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:00.703 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:00.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:00.796 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:00.797+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:00.877 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:00.877+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:00.880 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:00.880 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:00.880 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:00.880 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:00.880 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:00.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:00.978 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:00.978+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:01.061 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:01.061 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:01.080 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:01.079+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:01.181 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:01.181+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:01.201 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:01.202 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:01.282 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:01.282+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:01.338 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:01.338 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:01.383 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:01.382+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:01.483 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:01.483+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:01.553 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 100.573389878, 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.909608876999997, 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.906328247999994, 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.601551631999996, 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.600708534999995, 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.578173028999998, 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:59:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.576180609999994, 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:59:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:59:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.575648345000005, 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:59:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.558723435999994, 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.557886801999999, 2023-12-06T07:59:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.533439533999996, 2023-12-06T07:59:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:59:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.527537170000002, 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:59:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:59:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.522831174000004, 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:59:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:59:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:59:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.520529299000003, 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:59:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:59:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.520030976000001, 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:59:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.519577064999993, 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.518354665000004, 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 92.873188683999999, 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:59:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:59:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:59:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 92.865750269000003, 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 92.855676896999995, 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:59:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:01.633 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:01.635 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:01.635+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:01.736 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:01.736+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:01.762 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:01.762 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:01.837 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:01.837+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:01.938 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:01.938 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:01.939 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:01.939+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:01.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:02.040 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:02.040+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:02.117 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:02.117 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:02.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:02.118 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:02.118 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:02.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:02.141 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:02.141+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:02.241 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:02.242+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:02.291 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:02.291 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:02.291 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:02.291 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:02.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:02.342 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:02.342+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:02.443 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:02.443+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:02.468 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:02.468 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:02.468 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:02.468 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:02.468 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:02.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:02.544 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:02.544+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:02.608 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:02.608 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 82.031559180000002, 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:59:02.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:59:02.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:59:02.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:02.612 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:02.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:02.646 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:02.646+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:02.747 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:02.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:02.758 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:02.760 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:02.760 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:02.760 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:02.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:02.848 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:02.848+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:02.894 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:02.894 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:02.894 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:02.894 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:02.894 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:02.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:02.988 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:02.988+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:03.050 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:03.050+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:03.072 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:03.072 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:03.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:03.072 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:03.073 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:03.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:03.167 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:03.167+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:03.249 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:03.250 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:03.250 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:03.250 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:03.251 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:03.252+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:03.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:03.352 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:03.352+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:03.427 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:03.427 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:03.427 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:03.427 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:03.427 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:03.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:03.453 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:03.454+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:03.554 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:03.554+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:03.655 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:03.655+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 103.53000871, 2023-12-06T07:59:03.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:59:03.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 102.058895257, 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:59:03.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:59:03.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:59:03.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.828702878, 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:59:03.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:59:03.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:59:03.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:59:03.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:59:03.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.827995329, 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:03.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:59:03.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.827358335, 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:59:03.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.77029314000001, 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:59:03.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:59:03.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.768802369, 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.701 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:59:03.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.736452707, 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:59:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:59:03.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:59:03.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.72982193199999, 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.711 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:59:03.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.729171251, 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.716 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:03.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:59:03.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:59:03.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:03.721 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.72569856, 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:59:03.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:59:03.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:03.726 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.724559838, 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.723612141, 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:03.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.736 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.695248147, 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.66703862999999, 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:59:03.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:59:03.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.056840695999995, 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.747 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:59:03.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.056195856000002, 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.752 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:03.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:59:03.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:59:03.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.757 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.035729102000005, 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:03.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:59:03.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:59:03.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.763 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.034980185999999, 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:59:03.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:59:03.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:59:03.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.034338117000004, 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.769 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:59:03.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:03.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:03.774 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:03.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:03.776 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:03.775+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:03.877 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:03.877+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:03.944 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:03.944 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:03.944 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:03.945 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:03.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:03.998 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:03.998+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:04.079 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:04.079+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:04.084 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:04.085 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:04.085 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:04.085 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:04.085 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:04.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:04.180 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:04.180+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:04.265 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:04.266 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:04.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:04.266 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:04.266 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:04.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:04.281 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:04.281+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:04.382 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:04.382+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:04.442 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:04.442 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:04.443 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:04.443 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:04.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:04.482 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:04.482+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:04.582 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:04.582 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:04.582 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:04.582 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:04.582 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:04.583 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:04.583+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:04.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:04.683 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:04.684+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:04.724 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:04.724 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:04.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.307538377, 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:04.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:04.728 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:04.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:04.784 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:04.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:04.872 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:04.872 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:04.872 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:04.872 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:04.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:04.885 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:04.884+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:04.985 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:04.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:05.011 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.012 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:05.012 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:05.012 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:05.012 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:05.086 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:05.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:05.149 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.149 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:05.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:05.149 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:05.150 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:05.187 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:05.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:05.288 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:05.288+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:05.331 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.331 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:05.331 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:05.331 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:05.389 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:05.389+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:05.468 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.469 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:05.469 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:05.469 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:05.469 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:05.523 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:05.523+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:05.591 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:05.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 104.667063157, 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.651 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:59:05.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:59:05.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 104.003282156, 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.655 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 104.00000152699999, 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:05.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:59:05.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.695224911, 2023-12-06T07:59:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.694381814, 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.668 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:59:05.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:59:05.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:59:05.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.671846308, 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.672 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:05.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:59:05.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.669853889, 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:59:05.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:59:05.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:59:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.683 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.66932162400001, 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:05.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:59:05.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.65239671499999, 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:59:05.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:59:05.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 103.651560081, 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.696 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:59:05.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:05.701 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.627112813, 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:59:05.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:59:05.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.621210449, 2023-12-06T07:59:05.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:05.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.616504453, 2023-12-06T07:59:05.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:59:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:59:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:59:05.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.718 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.614202578, 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:59:05.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.613704255, 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:05.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:59:05.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:59:05.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:59:05.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:05.729 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.61325034399999, 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:59:05.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.612027944, 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:59:05.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:59:05.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.966861962999999, 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:59:05.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:59:05.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.959423548000004, 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:05.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.752 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.949350175999996, 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:59:05.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:59:05.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:59:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:05.758 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:05.761 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:05.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:05.862 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:05.861+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:05.930 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:05.930 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:05.930 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:05.930 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:05.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:05.964 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:05.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:06.064 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:06.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:06.068 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:06.068 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:06.069 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:06.069 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:06.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:06.167 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:06.165+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:06.246 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:06.247 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:06.247 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:06.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:06.265 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:06.265+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:06.366 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:06.365+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:06.427 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:06.427 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:06.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:06.466 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:06.466+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:06.568 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:06.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:06.602 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:06.602 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:06.603 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:06.603 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:06.603 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:06.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:06.669 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:06.668+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.165676339000001, 2023-12-06T07:59:06.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:06.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:59:06.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:06.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:06.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:06.746 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:06.746 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:06.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:06.795 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:06.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:06.870 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:06.870+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:06.891 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:06.891 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:06.891 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:06.891 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:06.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:06.971 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:06.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:07.068 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:07.068 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:07.072 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:07.072+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:07.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:07.173 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:07.173+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:07.246 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:07.246 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:07.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:07.275 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:07.274+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:07.375 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:07.375+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:07.424 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:07.424 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:07.424 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:07.424 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:07.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:07.477 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:07.476+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:07.561 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:07.562 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:07.562 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:07.562 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:07.562 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:07.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:07.615 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:07.615+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:07.681 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:07.679+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:07.756 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:07.756 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 107.627222733, 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.15610928, 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:59:07.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.92591690099999, 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:59:07.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:59:07.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:07.773 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.925209352, 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:59:07.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.778 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.92457235800001, 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:07.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:59:07.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:59:07.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.867507163, 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.784 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:59:07.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:59:07.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.86601639200001, 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.794 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.83366673, 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:59:07.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:59:07.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:59:07.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:59:07.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.827035955, 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:59:07.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:59:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:59:07.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.826385274, 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:59:07.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:59:07.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:59:07.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.822912583, 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:59:07.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.821773861, 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:59:07.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:59:07.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:59:07.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:59:07.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.821 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.820826164, 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:59:07.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:59:07.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:59:07.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.79246216999999, 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:59:07.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:59:07.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:59:07.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:59:07.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:07.833 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.764252653, 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:59:07.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:59:07.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:59:07.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:59:07.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.154054719000001, 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.839 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:59:07.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:59:07.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:59:07.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.153409878999994, 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.849 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.132943124999997, 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:59:07.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:59:07.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:59:07.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:59:07.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:59:07.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:59:07.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.132194209000005, 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.131552139999997, 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:59:07.865 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:07.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:07.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:59:07.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:07.871 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:07.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:07.874 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:07.872+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:07.974 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:07.974+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:08.043 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.043 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:08.043 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:08.043 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:08.076 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:08.075+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:08.177 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:08.177+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:08.180 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.180 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:08.180 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:08.180 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:08.180 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:08.277 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:08.277+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:08.355 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.356 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:08.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:08.356 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:08.356 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:08.405 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:08.405+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:08.479 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:08.478+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:08.531 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:08.531 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:08.585 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:08.585+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:08.668 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:08.668 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:08.685 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:08.684+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:08.783 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:08.783+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:08.849 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:08.849 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:08.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:08.849 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 89.432356631000005, 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:08.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:08.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:08.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:08.854 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:08.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:08.903 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:08.902+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:08.985 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:08.984+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:09.000 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:09.000 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:09.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:09.095 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:09.094+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:09.181 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:09.181 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:09.187 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:09.186+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:09.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:09.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:59:09.210 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:09.210 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 36, 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:58:48.535565+0000", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5804032, 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:09.211 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T07:59:09.212 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T07:59:09.213 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:59:09.214 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:59:09.215 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:59:09.216 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T07:59:09.217 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T07:59:09.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:59:09.219 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:59:09.219 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:59:09.219 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:59:09.219 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:59:09.219 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:59:09.220 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:59:09.221 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:59:09.221 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:59:09.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.221 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T07:59:09.221 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:59:09.222 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:59:09.223 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3.9800000190734863, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3.9800000190734863, 2023-12-06T07:59:09.224 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 3.5, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 3.5, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.225 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:09.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T07:59:09.228 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.229 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 30, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T07:59:09.230 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.231 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T07:59:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.233 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.235 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.236 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.237 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.240 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.242 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:09.244 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T07:59:09.245 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T07:59:09.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T07:59:09.247 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.249 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.251 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.252 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T07:59:09.253 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:09.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T07:59:09.256 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:09.257 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 24, 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.258 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.259 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T07:59:09.260 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:09.260 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.260 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T07:59:09.261 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.265 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T07:59:09.266 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:59:09.267 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.5", 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T07:59:09.268 INFO:teuthology.orchestra.run.smithi084.stdout: 6 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T07:59:09.269 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T07:59:09.270 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:09.271 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:09.273 INFO:tasks.thrashosds.thrasher:j is {'epoch': 36, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T07:58:48.535565+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.9800000190734863, 'score_stable': 3.9800000190734863, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 3.5, 'raw_score_stable': 3.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [1, 6]}, {'pgid': '2.6', 'osds': [7, 2]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T07:59:09.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:59:09.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:09.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:09.313 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:09.313+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:09.388 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:09.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:09.399 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:09.399 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:09.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:09.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:09.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:09.489 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:09.489+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:09.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:09.496 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:09.537 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:09.538 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:09.538 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:09.538 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:09.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:09.588 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:09.588 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:09.591 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:09.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:09.677 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:09.677 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:09.677 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:09.677 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:09.677 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:09.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:09.690 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:09.690+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:09.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:09.692 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:09.792 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:09.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:09.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:09.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:09.837 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:09.852 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T07:59:09.852 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T07:59:09.852 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T07:59:09.854 INFO:tasks.ceph.mon.b:Started 2023-12-06T07:59:09.855 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T07:59:09.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:09.879 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:09.879 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:09.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:09.893 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:09.893+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:09.909 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:09.909 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:09.993 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:09.993+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:10.028 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:59:10.029 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:59:10.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:10.040 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:10.042 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2023-12-06T07:59:09.819485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":219548,"kb_used_data":3376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755172,"statfs":{"total":773094113280,"available":772869296128,"internally_reserved":0,"allocated":3457024,"data_stored":1792480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002149"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:49.373621+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:58:49.373621+0000","last_peered":"2023-12-06T07:58:49.373621+0000","last_clean":"2023-12-06T07:58:49.373621+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:58:49.373621+0000","last_undegraded":"2023-12-06T07:58:49.373621+0000","last_fullsized":"2023-12-06T07:58:49.373621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-06T07:58:42.360755+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T07:58:42.360755+0000","last_peered":"2023-12-06T07:58:42.360755+0000","last_clean":"2023-12-06T07:58:42.360755+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T07:58:42.360755+0000","last_undegraded":"2023-12-06T07:58:42.360755+0000","last_fullsized":"2023-12-06T07:58:42.360755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:46.727937+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T07:58:46.727937+0000","last_peered":"2023-12-06T07:58:46.727937+0000","last_clean":"2023-12-06T07:58:46.727937+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T07:58:46.727937+0000","last_undegraded":"2023-12-06T07:58:46.727937+0000","last_fullsized":"2023-12-06T07:58:46.727937+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:47.054799+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:58:47.054799+0000","last_peered":"2023-12-06T07:58:47.054799+0000","last_clean":"2023-12-06T07:58:47.054799+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:58:47.054799+0000","last_undegraded":"2023-12-06T07:58:47.054799+0000","last_fullsized":"2023-12-06T07:58:47.054799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:49.370058+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:58:49.370058+0000","last_peered":"2023-12-06T07:58:49.370058+0000","last_clean":"2023-12-06T07:58:49.370058+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:58:49.370058+0000","last_undegraded":"2023-12-06T07:58:49.370058+0000","last_fullsized":"2023-12-06T07:58:49.370058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-06T07:58:42.356649+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:58:42.356649+0000","last_peered":"2023-12-06T07:58:42.356649+0000","last_clean":"2023-12-06T07:58:42.356649+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:58:42.356649+0000","last_undegraded":"2023-12-06T07:58:42.356649+0000","last_fullsized":"2023-12-06T07:58:42.356649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:49.361926+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:58:49.361926+0000","last_peered":"2023-12-06T07:58:49.361926+0000","last_clean":"2023-12-06T07:58:49.361926+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:58:49.361926+0000","last_undegraded":"2023-12-06T07:58:49.361926+0000","last_fullsized":"2023-12-06T07:58:49.361926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:49.366225+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:58:49.366225+0000","last_peered":"2023-12-06T07:58:49.366225+0000","last_clean":"2023-12-06T07:58:49.366225+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:58:49.366225+0000","last_undegraded":"2023-12-06T07:58:49.366225+0000","last_fullsized":"2023-12-06T07:58:49.366225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":132,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-06T07:58:46.723513+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:58:46.723513+0000","last_peered":"2023-12-06T07:58:46.723513+0000","last_clean":"2023-12-06T07:58:46.723513+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:58:46.723513+0000","last_undegraded":"2023-12-06T07:58:46.723513+0000","last_fullsized":"2023-12-06T07:58:46.723513+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476760,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":9.5909999999999993,"5min":9.5909999999999993,"15min":9.5909999999999993},"min":{"1min":5.3440000000000003,"5min":5.3440000000000003,"15min":5.3440000000000003},"max":{"1min":16.376999999999999,"5min":16.376999999999999,"15min":16.376999999999999},"last":6.9690000000000003},{"interface":"front","average":{"1min":8.609,"5min":8.609,"15min":8.609},"min":{"1min":4.8029999999999999,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":16.166,"5min":16.166,"15min":16.166},"last":9.423}]},{"osd":1,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":13.117000000000001,"5min":13.117000000000001,"15min":13.117000000000001},"min":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":29.213000000000001,"5min":29.213000000000001,"15min":29.213000000000001},"last":11.496},{"interface":"front","average":{"1min":14.73,"5min":14.73,"15min":14.73},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.337000000000003,"5min":39.337000000000003,"15min":39.337000000000003},"last":10.163}]},{"osd":2,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":12.635999999999999,"5min":12.635999999999999,"15min":12.635999999999999},"min":{"1min":7.7839999999999998,"5min":7.7839999999999998,"15min":7.7839999999999998},"max":{"1min":27.861000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":8.048},{"interface":"front","average":{"1min":14.220000000000001,"5min":14.220000000000001,"15min":14.220000000000001},"min":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":23.419,"5min":23.419,"15min":23.419},"last":18.536000000000001}]},{"osd":4,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":13.166,"5min":13.166,"15min":13.166},"min":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"max":{"1min":23.986000000000001,"5min":23.986000000000001,"15min":23.986000000000001},"last":10.794},{"interface":"front","average":{"1min":16.123999999999999,"5min":16.123999999999999,"15min":16.123999999999999},"min":{"1min":7.806,"5min":7.806,"15min":7.806},"max":{"1min":29.359000000000002,"5min":29.359000000000002,"15min":29.359000000000002},"last":17.734999999999999}]},{"osd":5,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":17.224,"5min":17.224,"15min":17.224},"min":{"1min":8.8580000000000005,"5min":8.8580000000000005,"15min":8.8580000000000005},"max":{"1min":28.75,"5min":28.75,"15min":28.75},"last":13.59},{"interface":"front","average":{"1min":17.396999999999998,"5min":17.396999999999998,"15min":17.396999999999998},"min":{"1min":10.679,"5min":10.679,"15min":10.679},"max":{"1min":38.670000000000002,"5min":38.670000000000002,"15min":38.670000000000002},"last":12.867000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":26.283999999999999,"5min":26.283999999999999,"15min":26.283999999999999},"last":15.994},{"interface":"front","average":{"1min":18.815999999999999,"5min":18.815999999999999,"15min":18.815999999999999},"min":{"1min":14.145,"5min":14.145,"15min":14.145},"max":{"1min":25.510000000000002,"5min":25.510000000000002,"15min":25.510000000000002},"last":19.332999999999998}]},{"osd":7,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":19.462,"5min":19.462,"15min":19.462},"min":{"1min":12.647,"5min":12.647,"15min":12.647},"max":{"1min":40.832000000000001,"5min":40.832000000000001,"15min":40.832000000000001},"last":12.214},{"interface":"front","average":{"1min":18.747,"5min":18.747,"15min":18.747},"min":{"1min":9.2330000000000005,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":39.331000000000003,"5min":39.331000000000003,"15min":39.331000000000003},"last":16.988}]}]},{"osd":2,"up_from":16,"seq":68719476762,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":10.497,"5min":10.497,"15min":10.497},"min":{"1min":5.7450000000000001,"5min":5.7450000000000001,"15min":5.7450000000000001},"max":{"1min":22.759,"5min":22.759,"15min":22.759},"last":14.541},{"interface":"front","average":{"1min":10.611000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"min":{"1min":5.0289999999999999,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":34.981000000000002,"5min":34.981000000000002,"15min":34.981000000000002},"last":6.5389999999999997}]},{"osd":1,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":22.494,"5min":22.494,"15min":22.494},"min":{"1min":6.7800000000000002,"5min":6.7800000000000002,"15min":6.7800000000000002},"max":{"1min":165.68600000000001,"5min":165.68600000000001,"15min":165.68600000000001},"last":15.276},{"interface":"front","average":{"1min":23.280999999999999,"5min":23.280999999999999,"15min":23.280999999999999},"min":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"max":{"1min":153.68199999999999,"5min":153.68199999999999,"15min":153.68199999999999},"last":9.2189999999999994}]},{"osd":3,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":11.272,"5min":11.272,"15min":11.272},"min":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":6.3869999999999996},"max":{"1min":35.935000000000002,"5min":35.935000000000002,"15min":35.935000000000002},"last":9.8949999999999996},{"interface":"front","average":{"1min":15.711,"5min":15.711,"15min":15.711},"min":{"1min":8.6560000000000006,"5min":8.6560000000000006,"15min":8.6560000000000006},"max":{"1min":28.687999999999999,"5min":28.687999999999999,"15min":28.687999999999999},"last":16.853999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":17.026,"5min":17.026,"15min":17.026},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":50.137999999999998,"5min":50.137999999999998,"15min":50.137999999999998},"last":12.119999999999999},{"interface":"front","average":{"1min":16.611000000000001,"5min":16.611000000000001,"15min":16.611000000000001},"min":{"1min":7.1429999999999998,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":46.325000000000003,"5min":46.325000000000003,"15min":46.325000000000003},"last":17.739000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":20.631,"5min":20.631,"15min":20.631},"min":{"1min":12.815,"5min":12.815,"15min":12.815},"max":{"1min":50.320999999999998,"5min":50.320999999999998,"15min":50.320999999999998},"last":19.280000000000001},{"interface":"front","average":{"1min":17.785,"5min":17.785,"15min":17.785},"min":{"1min":10.140000000000001,"5min":10.140000000000001,"15min":10.140000000000001},"max":{"1min":44.323,"5min":44.323,"15min":44.323},"last":7.9370000000000003}]},{"osd":6,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":20.587,"5min":20.587,"15min":20.587},"min":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":104.834,"5min":104.834,"15min":104.834},"last":13.491},{"interface":"front","average":{"1min":20.792999999999999,"5min":20.792999999999999,"15min":20.792999999999999},"min":{"1min":12.233000000000001,"5min":12.233000000000001,"15min":12.233000000000001},"max":{"1min":41.229999999999997,"5min":41.229999999999997,"15min":41.229999999999997},"last":20.155000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:58:22 2023","interfaces":[{"interface":"back","average":{"1min":19.943000000000001,"5min":19.943000000000001,"15min":19.943000000000001},"min":{"1min":7.8440000000000003,"5min":7.8440000000000003,"15min":7.8440000000000003},"max":{"1min":42.915999999999997,"5min":42.915999999999997,"15min":42.915999999999997},"last":11.255000000000001},{"interface":"front","average":{"1min":19.433,"5min":19.433,"15min":19.433},"min":{"1min":9.9339999999999993,"5min":9.9339999999999993,"15min":9.9339999999999993},"max":{"1min":42.506,"5min":42.506,"15min":42.506},"last":18.099}]}]},{"osd":1,"up_from":16,"seq":68719476762,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":522440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":13.68,"15min":13.68},"min":{"1min":5.7519999999999998,"5min":5.7519999999999998,"15min":5.7519999999999998},"max":{"1min":80.640000000000001,"5min":80.640000000000001,"15min":80.640000000000001},"last":5.0890000000000004},{"interface":"front","average":{"1min":11.662000000000001,"5min":11.662000000000001,"15min":11.662000000000001},"min":{"1min":6.1180000000000003,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.390000000000001,"5min":25.390000000000001,"15min":25.390000000000001},"last":7.9009999999999998}]},{"osd":2,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":21.216999999999999,"5min":21.216999999999999,"15min":21.216999999999999},"min":{"1min":7.6429999999999998,"5min":7.6429999999999998,"15min":7.6429999999999998},"max":{"1min":100.703,"5min":100.703,"15min":100.703},"last":14.798999999999999},{"interface":"front","average":{"1min":19.609000000000002,"5min":19.609000000000002,"15min":19.609000000000002},"min":{"1min":7.5179999999999998,"5min":7.5179999999999998,"15min":7.5179999999999998},"max":{"1min":95.960999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":6.532}]},{"osd":3,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":18.68,"5min":18.68,"15min":18.68},"min":{"1min":11.135999999999999,"5min":11.135999999999999,"15min":11.135999999999999},"max":{"1min":42.167999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":18.478999999999999},{"interface":"front","average":{"1min":13.971,"5min":13.971,"15min":13.971},"min":{"1min":6.8719999999999999,"5min":6.8719999999999999,"15min":6.8719999999999999},"max":{"1min":46.613,"5min":46.613,"15min":46.613},"last":9.173}]},{"osd":4,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":18.798999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"min":{"1min":8.6340000000000003,"5min":8.6340000000000003,"15min":8.6340000000000003},"max":{"1min":81.838999999999999,"5min":81.838999999999999,"15min":81.838999999999999},"last":13.843},{"interface":"front","average":{"1min":21.805,"5min":21.805,"15min":21.805},"min":{"1min":11.817,"5min":11.817,"15min":11.817},"max":{"1min":66.515000000000001,"5min":66.515000000000001,"15min":66.515000000000001},"last":16.300000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":16.632999999999999,"5min":16.632999999999999,"15min":16.632999999999999},"min":{"1min":8.5190000000000001,"5min":8.5190000000000001,"15min":8.5190000000000001},"max":{"1min":56.235999999999997,"5min":56.235999999999997,"15min":56.235999999999997},"last":9.7750000000000004},{"interface":"front","average":{"1min":19.995999999999999,"5min":19.995999999999999,"15min":19.995999999999999},"min":{"1min":10.754,"5min":10.754,"15min":10.754},"max":{"1min":82.381,"5min":82.381,"15min":82.381},"last":15.534000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":18.928999999999998,"5min":18.928999999999998,"15min":18.928999999999998},"min":{"1min":8.7330000000000005,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":58.619,"5min":58.619,"15min":58.619},"last":11.069000000000001},{"interface":"front","average":{"1min":21.527000000000001,"5min":21.527000000000001,"15min":21.527000000000001},"min":{"1min":9.6229999999999993,"5min":9.6229999999999993,"15min":9.6229999999999993},"max":{"1min":83.632000000000005,"5min":83.632000000000005,"15min":83.632000000000005},"last":16.914000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:58:23 2023","interfaces":[{"interface":"back","average":{"1min":23.035,"5min":23.035,"15min":23.035},"min":{"1min":15.109999999999999,"5min":15.109999999999999,"15min":15.109999999999999},"max":{"1min":62.006999999999998,"5min":62.006999999999998,"15min":62.006999999999998},"last":19.265999999999998},{"interface":"front","average":{"1min":20.998000000000001,"5min":20.998000000000001,"15min":20.998000000000001},"min":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"max":{"1min":60.698999999999998,"5min":60.698999999999998,"15min":60.698999999999998},"last":10.445}]}]},{"osd":0,"up_from":16,"seq":68719476763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":20.725000000000001,"5min":20.725000000000001,"15min":20.725000000000001},"min":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"max":{"1min":252.88399999999999,"5min":252.88399999999999,"15min":252.88399999999999},"last":7.7279999999999998},{"interface":"front","average":{"1min":21.887,"5min":21.887,"15min":21.887},"min":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":274.91500000000002,"5min":274.91500000000002,"15min":274.91500000000002},"last":8.7479999999999993}]},{"osd":2,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":17.760999999999999,"5min":17.760999999999999,"15min":17.760999999999999},"min":{"1min":6.625,"5min":6.625,"15min":6.625},"max":{"1min":82.866,"5min":82.866,"15min":82.866},"last":18.975000000000001},{"interface":"front","average":{"1min":16.579999999999998,"5min":16.579999999999998,"15min":16.579999999999998},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":85.991,"5min":85.991,"15min":85.991},"last":11.789999999999999}]},{"osd":3,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":15.461,"5min":15.461,"15min":15.461},"min":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"max":{"1min":76.593000000000004,"5min":76.593000000000004,"15min":76.593000000000004},"last":8.0269999999999992},{"interface":"front","average":{"1min":21.064,"5min":21.064,"15min":21.064},"min":{"1min":9.3710000000000004,"5min":9.3710000000000004,"15min":9.3710000000000004},"max":{"1min":85.162000000000006,"5min":85.162000000000006,"15min":85.162000000000006},"last":13.148999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":17.577999999999999,"5min":17.577999999999999,"15min":17.577999999999999},"min":{"1min":7.516,"5min":7.516,"15min":7.516},"max":{"1min":65.349000000000004,"5min":65.349000000000004,"15min":65.349000000000004},"last":13.771000000000001},{"interface":"front","average":{"1min":14.099,"5min":14.099,"15min":14.099},"min":{"1min":8.218,"5min":8.218,"15min":8.218},"max":{"1min":70.085999999999999,"5min":70.085999999999999,"15min":70.085999999999999},"last":9.9719999999999995}]},{"osd":5,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":28.373000000000001,"5min":28.373000000000001,"15min":28.373000000000001},"min":{"1min":11.129,"5min":11.129,"15min":11.129},"max":{"1min":144.60499999999999,"5min":144.60499999999999,"15min":144.60499999999999},"last":15.087},{"interface":"front","average":{"1min":26.704000000000001,"5min":26.704000000000001,"15min":26.704000000000001},"min":{"1min":10.827,"5min":10.827,"15min":10.827},"max":{"1min":142.25,"5min":142.25,"15min":142.25},"last":14.397}]},{"osd":6,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":18.434999999999999,"5min":18.434999999999999,"15min":18.434999999999999},"min":{"1min":9.5540000000000003,"5min":9.5540000000000003,"15min":9.5540000000000003},"max":{"1min":75.471999999999994,"5min":75.471999999999994,"15min":75.471999999999994},"last":10.544},{"interface":"front","average":{"1min":24.48,"5min":24.48,"15min":24.48},"min":{"1min":11.095000000000001,"5min":11.095000000000001,"15min":11.095000000000001},"max":{"1min":84.617000000000004,"5min":84.617000000000004,"15min":84.617000000000004},"last":18.024000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":22.231000000000002,"5min":22.231000000000002,"15min":22.231000000000002},"min":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":84.031000000000006,"5min":84.031000000000006,"15min":84.031000000000006},"last":15.766999999999999},{"interface":"front","average":{"1min":19.312999999999999,"5min":19.312999999999999,"15min":19.312999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":77.974000000000004,"5min":77.974000000000004,"15min":77.974000000000004},"last":11.101000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:26 2023","interfaces":[{"interface":"back","average":{"1min":9.718,"5min":9.718,"15min":9.718},"min":{"1min":4.351,"5min":4.351,"15min":4.351},"max":{"1min":23.677,"5min":23.677,"15min":23.677},"last":4.2380000000000004},{"interface":"front","average":{"1min":10.292,"5min":10.292,"15min":10.292},"min":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"max":{"1min":22.443999999999999,"5min":22.443999999999999,"15min":22.443999999999999},"last":10.776}]},{"osd":1,"last update":"Wed Dec 6 07:58:26 2023","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":12.031000000000001,"15min":12.031000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":6.4900000000000002},"max":{"1min":35.524000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":9.2390000000000008},{"interface":"front","average":{"1min":14.053000000000001,"5min":14.053000000000001,"15min":14.053000000000001},"min":{"1min":5.6269999999999998,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":31.027000000000001,"5min":31.027000000000001,"15min":31.027000000000001},"last":11.800000000000001}]},{"osd":2,"last update":"Wed Dec 6 07:58:26 2023","interfaces":[{"interface":"back","average":{"1min":12.016999999999999,"5min":12.016999999999999,"15min":12.016999999999999},"min":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"max":{"1min":16.321000000000002,"5min":16.321000000000002,"15min":16.321000000000002},"last":12.611000000000001},{"interface":"front","average":{"1min":14.625999999999999,"5min":14.625999999999999,"15min":14.625999999999999},"min":{"1min":8.3200000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":24.292999999999999,"5min":24.292999999999999,"15min":24.292999999999999},"last":13.456}]},{"osd":3,"last update":"Wed Dec 6 07:58:26 2023","interfaces":[{"interface":"back","average":{"1min":11.484999999999999,"5min":11.484999999999999,"15min":11.484999999999999},"min":{"1min":6.843,"5min":6.843,"15min":6.843},"max":{"1min":25.640999999999998,"5min":25.640999999999998,"15min":25.640999999999998},"last":8.016},{"interface":"front","average":{"1min":14.065,"5min":14.065,"15min":14.065},"min":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"max":{"1min":22.236999999999998,"5min":22.236999999999998,"15min":22.236999999999998},"last":15.923999999999999}]},{"osd":5,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":17.099,"5min":17.099,"15min":17.099},"min":{"1min":9.4529999999999994,"5min":9.4529999999999994,"15min":9.4529999999999994},"max":{"1min":50.417999999999999,"5min":50.417999999999999,"15min":50.417999999999999},"last":14.266999999999999},{"interface":"front","average":{"1min":17.669,"5min":17.669,"15min":17.669},"min":{"1min":9.9399999999999995,"5min":9.9399999999999995,"15min":9.9399999999999995},"max":{"1min":48.837000000000003,"5min":48.837000000000003,"15min":48.837000000000003},"last":16.739000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":22.454000000000001,"5min":22.454000000000001,"15min":22.454000000000001},"min":{"1min":10.954000000000001,"5min":10.954000000000001,"15min":10.954000000000001},"max":{"1min":93.073999999999998,"5min":93.073999999999998,"15min":93.073999999999998},"last":14.815},{"interface":"front","average":{"1min":18.713000000000001,"5min":18.713000000000001,"15min":18.713000000000001},"min":{"1min":8.0779999999999994,"5min":8.0779999999999994,"15min":8.0779999999999994},"max":{"1min":91.075999999999993,"5min":91.075999999999993,"15min":91.075999999999993},"last":7.2949999999999999}]},{"osd":7,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":16.367999999999999,"5min":16.367999999999999,"15min":16.367999999999999},"min":{"1min":11.583,"5min":11.583,"15min":11.583},"max":{"1min":27.434000000000001,"5min":27.434000000000001,"15min":27.434000000000001},"last":9.8360000000000003},{"interface":"front","average":{"1min":16.731000000000002,"5min":16.731000000000002,"15min":16.731000000000002},"min":{"1min":12.539,"5min":12.539,"15min":12.539},"max":{"1min":25.594999999999999,"5min":25.594999999999999,"15min":25.594999999999999},"last":17.507000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":10.311,"15min":10.311},"min":{"1min":4.0810000000000004,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":70.965000000000003,"5min":70.965000000000003,"15min":70.965000000000003},"last":6.758},{"interface":"front","average":{"1min":12.337999999999999,"5min":12.337999999999999,"15min":12.337999999999999},"min":{"1min":5.4329999999999998,"5min":5.4329999999999998,"15min":5.4329999999999998},"max":{"1min":73.582999999999998,"5min":73.582999999999998,"15min":73.582999999999998},"last":11.691000000000001}]},{"osd":1,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":19.559000000000001,"5min":19.559000000000001,"15min":19.559000000000001},"min":{"1min":4.383,"5min":4.383,"15min":4.383},"max":{"1min":217.595,"5min":217.595,"15min":217.595},"last":9.7240000000000002},{"interface":"front","average":{"1min":23.288,"5min":23.288,"15min":23.288},"min":{"1min":8.0909999999999993,"5min":8.0909999999999993,"15min":8.0909999999999993},"max":{"1min":284.96499999999997,"5min":284.96499999999997,"15min":284.96499999999997},"last":7.9950000000000001}]},{"osd":2,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":13.882,"5min":13.882,"15min":13.882},"min":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":46.024999999999999,"5min":46.024999999999999,"15min":46.024999999999999},"last":19.815000000000001},{"interface":"front","average":{"1min":16.984000000000002,"5min":16.984000000000002,"15min":16.984000000000002},"min":{"1min":9.5920000000000005,"5min":9.5920000000000005,"15min":9.5920000000000005},"max":{"1min":46.966999999999999,"5min":46.966999999999999,"15min":46.966999999999999},"last":16.931999999999999}]},{"osd":3,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":15.846,"5min":15.846,"15min":15.846},"min":{"1min":7.2190000000000003,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":51.031999999999996,"5min":51.031999999999996,"15min":51.031999999999996},"last":15.092000000000001},{"interface":"front","average":{"1min":24.356000000000002,"5min":24.356000000000002,"15min":24.356000000000002},"min":{"1min":7.6139999999999999,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":189.84999999999999,"5min":189.84999999999999,"15min":189.84999999999999},"last":12.394}]},{"osd":4,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":15.884,"5min":15.884,"15min":15.884},"min":{"1min":10.314,"5min":10.314,"15min":10.314},"max":{"1min":25.013000000000002,"5min":25.013000000000002,"15min":25.013000000000002},"last":12.984999999999999},{"interface":"front","average":{"1min":14.630000000000001,"5min":14.630000000000001,"15min":14.630000000000001},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":26.831,"5min":26.831,"15min":26.831},"last":17.905999999999999}]},{"osd":5,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":18.923999999999999,"5min":18.923999999999999,"15min":18.923999999999999},"min":{"1min":11.26,"5min":11.26,"15min":11.26},"max":{"1min":50.945,"5min":50.945,"15min":50.945},"last":13.617000000000001},{"interface":"front","average":{"1min":14.372999999999999,"5min":14.372999999999999,"15min":14.372999999999999},"min":{"1min":7.3090000000000002,"5min":7.3090000000000002,"15min":7.3090000000000002},"max":{"1min":51.948,"5min":51.948,"15min":51.948},"last":10.487}]},{"osd":7,"last update":"Wed Dec 6 07:58:19 2023","interfaces":[{"interface":"back","average":{"1min":15.760999999999999,"5min":15.760999999999999,"15min":15.760999999999999},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":31.353999999999999,"5min":31.353999999999999,"15min":31.353999999999999},"last":18.931999999999999},{"interface":"front","average":{"1min":19.503,"5min":19.503,"15min":19.503},"min":{"1min":13.362,"5min":13.362,"15min":13.362},"max":{"1min":28.082000000000001,"5min":28.082000000000001,"15min":28.082000000000001},"last":15.84}]}]},{"osd":5,"up_from":14,"seq":60129542169,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":522440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":9.1370000000000005,"5min":9.1370000000000005,"15min":9.1370000000000005},"min":{"1min":4.6470000000000002,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":23.09,"5min":23.09,"15min":23.09},"last":5.7779999999999996},{"interface":"front","average":{"1min":9.6340000000000003,"5min":9.6340000000000003,"15min":9.6340000000000003},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":20.728000000000002,"5min":20.728000000000002,"15min":20.728000000000002},"last":7.8289999999999997}]},{"osd":1,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":13.824999999999999,"5min":13.824999999999999,"15min":13.824999999999999},"min":{"1min":6.718,"5min":6.718,"15min":6.718},"max":{"1min":67.347999999999999,"5min":67.347999999999999,"15min":67.347999999999999},"last":12.193},{"interface":"front","average":{"1min":12.759,"5min":12.759,"15min":12.759},"min":{"1min":5.8090000000000002,"5min":5.8090000000000002,"15min":5.8090000000000002},"max":{"1min":39.052,"5min":39.052,"15min":39.052},"last":10.199}]},{"osd":2,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":12.122999999999999,"5min":12.122999999999999,"15min":12.122999999999999},"min":{"1min":6.7279999999999998,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":27.73,"5min":27.73,"15min":27.73},"last":9.3469999999999995},{"interface":"front","average":{"1min":16.050999999999998,"5min":16.050999999999998,"15min":16.050999999999998},"min":{"1min":8.9350000000000005,"5min":8.9350000000000005,"15min":8.9350000000000005},"max":{"1min":34.585000000000001,"5min":34.585000000000001,"15min":34.585000000000001},"last":15.012}]},{"osd":3,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":15.535,"5min":15.535,"15min":15.535},"min":{"1min":6.9450000000000003,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":30.355,"5min":30.355,"15min":30.355},"last":28.518000000000001},{"interface":"front","average":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"min":{"1min":5.9580000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":31.285,"5min":31.285,"15min":31.285},"last":29.937999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":29.442,"5min":29.442,"15min":29.442},"min":{"1min":8.5150000000000006,"5min":8.5150000000000006,"15min":8.5150000000000006},"max":{"1min":334.86500000000001,"5min":334.86500000000001,"15min":334.86500000000001},"last":17.228999999999999},{"interface":"front","average":{"1min":29.279,"5min":29.279,"15min":29.279},"min":{"1min":7.431,"5min":7.431,"15min":7.431},"max":{"1min":331.57900000000001,"5min":331.57900000000001,"15min":331.57900000000001},"last":11.395}]},{"osd":6,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":20.074999999999999,"5min":20.074999999999999,"15min":20.074999999999999},"min":{"1min":10.382999999999999,"5min":10.382999999999999,"15min":10.382999999999999},"max":{"1min":46.034999999999997,"5min":46.034999999999997,"15min":46.034999999999997},"last":16.286999999999999},{"interface":"front","average":{"1min":18.885999999999999,"5min":18.885999999999999,"15min":18.885999999999999},"min":{"1min":12.143000000000001,"5min":12.143000000000001,"15min":12.143000000000001},"max":{"1min":35.680999999999997,"5min":35.680999999999997,"15min":35.680999999999997},"last":13.553000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:58:20 2023","interfaces":[{"interface":"back","average":{"1min":31.326000000000001,"5min":31.326000000000001,"15min":31.326000000000001},"min":{"1min":8.4239999999999995,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":332.322,"5min":332.322,"15min":332.322},"last":14.288},{"interface":"front","average":{"1min":34.549999999999997,"5min":34.549999999999997,"15min":34.549999999999997},"min":{"1min":11.305999999999999,"5min":11.305999999999999,"15min":11.305999999999999},"max":{"1min":325.93599999999998,"5min":325.93599999999998,"15min":325.93599999999998},"last":18.032}]}]},{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":124600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":9.5600000000000005,"5min":9.5600000000000005,"15min":9.5600000000000005},"min":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"max":{"1min":20.558,"5min":20.558,"15min":20.558},"last":7.516},{"interface":"front","average":{"1min":11.984,"5min":11.984,"15min":11.984},"min":{"1min":6.5099999999999998,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":20.893999999999998,"5min":20.893999999999998,"15min":20.893999999999998},"last":13.112}]},{"osd":1,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":16.837,"5min":16.837,"15min":16.837},"min":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":51.890000000000001,"5min":51.890000000000001,"15min":51.890000000000001},"last":11.673},{"interface":"front","average":{"1min":8.3919999999999995,"5min":8.3919999999999995,"15min":8.3919999999999995},"min":{"1min":4.2110000000000003,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":35.075000000000003,"5min":35.075000000000003,"15min":35.075000000000003},"last":6.9779999999999998}]},{"osd":2,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":12.608000000000001,"5min":12.608000000000001,"15min":12.608000000000001},"min":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":35.225999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":7.9820000000000002},{"interface":"front","average":{"1min":11.156000000000001,"5min":11.156000000000001,"15min":11.156000000000001},"min":{"1min":7.2030000000000003,"5min":7.2030000000000003,"15min":7.2030000000000003},"max":{"1min":19.378,"5min":19.378,"15min":19.378},"last":10.906000000000001}]},{"osd":3,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":30.164000000000001,"5min":30.164000000000001,"15min":30.164000000000001},"min":{"1min":7.3140000000000001,"5min":7.3140000000000001,"15min":7.3140000000000001},"max":{"1min":228.17099999999999,"5min":228.17099999999999,"15min":228.17099999999999},"last":12.502000000000001},{"interface":"front","average":{"1min":33.161999999999999,"5min":33.161999999999999,"15min":33.161999999999999},"min":{"1min":9.7769999999999992,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":270.13,"5min":270.13,"15min":270.13},"last":14.433999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:58:21 2023","interfaces":[{"interface":"back","average":{"1min":17.405999999999999,"5min":17.405999999999999,"15min":17.405999999999999},"min":{"1min":7.101,"5min":7.101,"15min":7.101},"max":{"1min":86.980999999999995,"5min":86.980999999999995,"15min":86.980999999999995},"last":18.565000000000001},{"interface":"front","average":{"1min":20.826000000000001,"5min":20.826000000000001,"15min":20.826000000000001},"min":{"1min":8.8859999999999992,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":99.021000000000001,"5min":99.021000000000001,"15min":99.021000000000001},"last":15.207000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:58:17 2023","interfaces":[{"interface":"back","average":{"1min":22.213000000000001,"5min":22.213000000000001,"15min":22.213000000000001},"min":{"1min":11.497999999999999,"5min":11.497999999999999,"15min":11.497999999999999},"max":{"1min":97.191000000000003,"5min":97.191000000000003,"15min":97.191000000000003},"last":16.431999999999999},{"interface":"front","average":{"1min":21.489999999999998,"5min":21.489999999999998,"15min":21.489999999999998},"min":{"1min":9.5519999999999996,"5min":9.5519999999999996,"15min":9.5519999999999996},"max":{"1min":64.048000000000002,"5min":64.048000000000002,"15min":64.048000000000002},"last":19.670999999999999}]},{"osd":6,"last update":"Wed Dec 6 07:58:17 2023","interfaces":[{"interface":"back","average":{"1min":17.614999999999998,"5min":17.614999999999998,"15min":17.614999999999998},"min":{"1min":8.7230000000000008,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":74.064999999999998,"5min":74.064999999999998,"15min":74.064999999999998},"last":19.253},{"interface":"front","average":{"1min":22.631,"5min":22.631,"15min":22.631},"min":{"1min":13.166,"5min":13.166,"15min":13.166},"max":{"1min":92.477999999999994,"5min":92.477999999999994,"15min":92.477999999999994},"last":15.811}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:59:10.044 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [0, 1, 3, 4] 2023-12-06T07:59:10.044 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.6', '0', '1', '3', '4'] 2023-12-06T07:59:10.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.6 0 1 3 4 2023-12-06T07:59:10.073 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.073 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:10.095 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:10.095+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:10.195 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:10.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:10.216 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.216 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:10.297 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:10.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:10.355 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.355 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:10.374 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T07:59:10.433 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:10.433+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:10.498 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:10.498+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:10.538 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.538 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:10.568 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:10.600 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:10.599+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:10.682 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.682 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:10.737 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:10.734+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:10.803 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:10.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:10.865 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:10.866 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:10.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:10.905 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:10.904+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:11.004 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.004 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.004 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:11.004+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:11.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:11.106 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:11.105+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:11.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.152 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.152 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.6 pg_upmap_items mapping to [0->1,3->4] 2023-12-06T07:59:11.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:11.207 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:11.206+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:11.302 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.302 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.308 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:11.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:11.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:11.409 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:11.408+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:11.440 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.440 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:11.509 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:11.509+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:11.580 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.580 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:11.631 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:11.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:11.711 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:11.711+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:11.764 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.764 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:11.817 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:11.816+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:11.913 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:11.913+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:11.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:12.038 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:12.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:12.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:59:12.115 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:12.115+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:12.123 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:12.123 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:12.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:12.216 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:12.216+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:12.309 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:12.309 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:12.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:12.318 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:12.317+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:12.362 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:12.362 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:12.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T07:59:12.437 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:12.437+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:12.450 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:12.451 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:12.500 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:12.500 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:12.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:12.519 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:12.518+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:12.544 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:12.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:12.620 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:12.620+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:12.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:12.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:12.688 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:12.688 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:12.688 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:12.688 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:12.688 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:12.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:12.740 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:12.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:12.741 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:12.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:12.823 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:12.822+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:12.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:12.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.470535663999996, 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:12.888 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:12.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:12.892 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:12.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:12.924 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:12.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:12.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:12.930 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:13.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:13.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:13.024 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:13.024+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:13.030 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.030 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:13.030 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:13.030 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:13.126 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:13.125+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.210 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:13.210 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:13.210 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:13.226 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:13.226+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:13.327 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:13.326+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:13.392 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.392 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:13.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:13.394 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:13.394 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:13.428 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:13.428+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:13.530 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:13.529+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:13.571 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.571 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:13.572 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:13.572 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:13.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:13.631 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:13.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:13.709 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:13.709 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:13.763 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:13.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:13.832 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:13.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 112.907615773, 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:59:13.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 112.243834772, 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:59:13.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:59:13.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:59:13.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 112.240554143, 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:59:13.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:13.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:59:13.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:59:13.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.935777527, 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.909 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.93493443, 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.912398924, 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:13.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:13.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:59:13.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:59:13.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:59:13.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:59:13.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:59:13.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:59:13.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.910406505, 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:59:13.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:59:13.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:59:13.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.90987423999999, 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.928 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:59:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:59:13.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:59:13.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.892949331, 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.932 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:59:13.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:59:13.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:59:13.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:13.936 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.892112697, 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:59:13.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.940 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.867665429, 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:59:13.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:59:13.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.944 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.86176306500001, 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:13.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:59:13.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:59:13.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.948 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.85705706900001, 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:13.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:59:13.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:59:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.85475519400001, 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.854256871, 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.85380296, 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:59:13.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:59:13.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:59:13.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:59:13.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 110.85258056000001, 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:59:13.966 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:59:13.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.207414579, 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:13.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:59:13.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:59:13.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:59:13.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.19997616400001, 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:13.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:59:13.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:59:13.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:59:13.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.189902792, 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:13.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:59:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:59:13.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:59:13.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.984 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.984 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.984 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:13.984 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:13.984 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:13.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:13.986 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:13.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:14.036 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T07:59:14.086 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:14.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:14.166 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.167 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:14.167 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:14.167 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:14.221 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:14.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:14.253 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:14.288 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:14.288+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:14.306 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.306 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:14.307 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:14.307 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:14.307 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:14.390 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:14.390+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:14.486 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.486 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:14.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:14.486 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:14.486 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.491 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:14.490+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:14.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:14.591 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:14.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:14.665 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.665 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:14.665 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:14.665 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:14.693 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:14.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:14.793 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:14.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:14.848 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.848 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:14.848 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:14.848 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:14.849 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:14.902 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:14.901+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:14.989 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:14.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.412916151999994, 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.990 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:59:14.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:59:14.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:14.993 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:14.995 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:14.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:14.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:15.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:59:15.096 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:15.095+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:15.133 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:15.133 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:15.133 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:15.133 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:15.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:15.197 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:15.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:15.298 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:15.297+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:15.325 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:15.325 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:15.325 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:15.325 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:15.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:15.399 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:15.399+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:15.463 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:15.463 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:15.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:15.513 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:15.513+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:15.601 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:15.600+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:15.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:15.614 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:15.666 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:15.666 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:15.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:15.701 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:15.701+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:15.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:15.718 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:15.802 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:15.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:15.805 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:15.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:15.850 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:15.851 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:15.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:15.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:15.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:15.903 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:15.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:15.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:15.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:15.994 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:15.995 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:16.004 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:16.004+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:16.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T07:59:16.068 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:16.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:16.109 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:16.109+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:16.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:16.155 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:16.173 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:59:16.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:59:16.173 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-06T07:59:16.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-06T07:59:16.181 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:16.181 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:16.231 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:16.231+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:16.307 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:16.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:16.368 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:16.368 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:16.420 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:16.420+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:16.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-06T07:59:16.508 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:16.508+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:16.605 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:16.605 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.609 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:16.608+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:16.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:16.710 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:16.709+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:16.760 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:16.761 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-06T07:59:16.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:16.813 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:16.812+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:16.833 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:15.681 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:16.837 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:15.685 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:16.860 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:15.707 94560== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:16.906 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:43.692 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:16.910 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:43.700 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:16.911 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:16.911+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:16.953 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:16.953 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:16.963 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:16.993 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:43.783 97394== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:17.037 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:17.037+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:17.113 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:17.113+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:17.133 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:17.134 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:17.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:17.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-06T07:59:17.214 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:17.214+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:17.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:17.272 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:17.272 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:17.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:17.323 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:17.323+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:17.416 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:17.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:17.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2023-12-06T07:59:17.469 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:17.469 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:17.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:17.517 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:17.517+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:17.618 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:17.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:17.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2023-12-06T07:59:17.654 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:17.655 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:17.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:17.748 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:17.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:17.821 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:17.820+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:17.835 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:17.835 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:17.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:17.858 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T07:59:17.914 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:07.998 95130== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:17.927 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:17.926+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:17.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2023-12-06T07:59:17.975 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:18.015 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.015 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:18.030 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:18.030+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:18.122 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:18.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:18.151 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:08.235 95130== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:18.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:59:18.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2023-12-06T07:59:18.234 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:18.234+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:18.238 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.238 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:18.325 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:18.324+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:18.378 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.379 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:18.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:18.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:18.430 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:18.430+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:18.511 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:18.512 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:18.526 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:18.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:18.558 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.558 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:18.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:18.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:18.626 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:18.626+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:18.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:18.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:18.718 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.718 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.727 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:18.727+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:18.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:18.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:18.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:18.829 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:18.829+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:18.862 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:18.862 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:18.918 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:18.918 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:18.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:18.929 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:18.929+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:18.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:18.952 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:19.030 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:19.030+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:19.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:19.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:19.131 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:19.130+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:19.143 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:19.143 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:19.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:19.233 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:19.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:19.334 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:19.334+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:19.342 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.764878252000003, 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:59:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:59:19.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:19.346 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:19.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:19.434 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:19.434+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:19.479 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:19.479 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:19.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:19.574 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:19.574+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:19.635 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:19.635+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:19.661 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:19.661 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:19.662 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:19.662 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:19.662 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:19.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:19.736 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:19.735+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:19.837 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:19.837+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:19.840 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:19.840 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:19.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:19.841 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:19.841 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:19.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:19.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T07:59:19.938 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:19.938+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:19.982 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:19.983 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:19.983 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:19.983 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:19.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:20.039 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:20.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:20.155 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:20.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:20.169 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:20.169 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:20.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:20.241 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:20.240+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:20.341 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:20.341+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:20.404 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.275729223, 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:59:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:59:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.80461577, 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:59:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:59:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:59:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.574423391, 2023-12-06T07:59:20.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:20.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.573715842, 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:59:20.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:59:20.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:59:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.432 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.57307884799999, 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:59:20.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:59:20.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:59:20.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.516013653, 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:59:20.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:59:20.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:59:20.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.51452288199999, 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.447 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:59:20.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:59:20.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:59:20.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.48217322000001, 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.453 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:59:20.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.475542445, 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:59:20.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:59:20.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:59:20.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:59:20.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:59:20.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.467 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.47489176400001, 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:20.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:59:20.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:59:20.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.47141907300001, 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:20.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:20.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:59:20.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.470280351, 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.482 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:20.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:59:20.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:59:20.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:59:20.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:59:20.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.469332654, 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:59:20.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:59:20.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:59:20.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:20.494 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.44096866, 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:59:20.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:59:20.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.412759143, 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.802561209, 2023-12-06T07:59:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:59:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.801916369, 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:59:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:59:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:59:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:59:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.781449615, 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.78070069899999, 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:59:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:59:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:59:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.78005863, 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:59:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:59:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:59:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:20.533 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:20.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:20.535 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:20.535+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:20.635 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:20.635+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:20.665 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:20.665 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:20.665 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:20.665 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:20.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:20.761 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:20.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:20.837 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:20.837+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:20.847 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:20.847 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:20.847 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:20.847 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:20.847 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:20.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:20.942 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:20.942+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:21.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:21.065 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T07:59:21.066 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:21.066 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:21.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:21.066 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:21.066 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:21.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:21.067 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:21.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:21.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:59:21.167 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:21.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:21.244 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:21.244 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:21.244 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:21.244 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:21.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:21.267 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:21.267+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:21.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:21.363 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:21.369 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:21.368+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:21.425 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:21.425 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:21.425 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:21.426 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:21.426 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:21.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:21.479 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:21.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:21.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:21.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:21.571 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:21.571+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:21.572 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:21.572 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:21.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 102.155282603, 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:21.576 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:21.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:21.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:21.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:21.682 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:21.683 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:21.683 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:21.682+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:21.726 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:21.726 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:21.726 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:21.726 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:21.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:21.773 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:21.773+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:21.786 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:21.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:21.853 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:21.885 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:21.886 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:21.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:21.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:21.892 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:21.891+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:21.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:21.975 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:21.975+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:21.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:21.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:22.024 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.024 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:22.068 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:22.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:22.076 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:22.076+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:22.161 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.161 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:22.215 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:22.214+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:22.278 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:22.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:22.304 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.304 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:22.378 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:22.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:22.479 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:22.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:22.482 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.482 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:22.581 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:22.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:22.711 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:22.711+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:22.782 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:22.782+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:22.794 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.794 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:22.883 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:22.883+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:22.971 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:22.971 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:22.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:23.024 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:23.023+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:23.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T07:59:23.085 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:23.085+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:23.106 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.107 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:23.187 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:23.186+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:23.288 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:23.287+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:23.289 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.289 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:23.388 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:23.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:23.451 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:59:23.451 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:59:23.466 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.466 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:23.517 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:23.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:23.590 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:23.589+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:23.605 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.605 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:23.691 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:23.691+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:23.748 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.748 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:23.802 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:23.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:23.885 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:23.885 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:23.892 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:23.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:23.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:23.994 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:23.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:24.064 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:24.065 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:24.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:24.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:59:24.116 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:24.116+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:24.195 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:24.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T07:59:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:24.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:24.297 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:24.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:24.386 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:24.397 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:24.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:24.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:24.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:24.497 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:24.497+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:24.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:24.560 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.43346771, 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:59:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:59:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:59:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.962354257, 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:59:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:59:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.732161878, 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:59:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:59:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.731454329, 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:59:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:59:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.730817335, 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:59:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.67375214, 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:59:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:59:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.672261369, 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:59:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:59:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:59:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.639911707, 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:59:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.63328093200001, 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:59:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:59:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.63263025099999, 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:59:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:59:24.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:59:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.62915756, 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:59:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.628018838, 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:59:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:59:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.627071141, 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:59:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:59:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.598707147, 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:59:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:59:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.57049763000001, 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:59:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:59:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.96029969600001, 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:59:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:59:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.959654856, 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:59:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.939188102, 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:59:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:59:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.938439186, 2023-12-06T07:59:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:59:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.937797117, 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:59:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:59:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:59:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:59:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:24.684 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:24.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:24.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:24.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:24.689 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:24.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:24.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:24.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:24.786 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:24.786+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:24.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:24.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:24.855 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:24.855 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:24.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:24.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:24.888 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:24.887+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:24.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:24.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:24.988 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:24.988+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:24.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:24.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:25.045 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:25.045 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:25.045 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:25.045 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:25.045 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:25.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:25.099 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:25.099+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:25.184 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:25.185 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:25.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:25.185 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:25.185 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:25.190 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:25.190+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:25.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:25.291 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:25.291+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:25.392 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:25.392+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:25.402 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:25.402 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:25.402 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:25.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:25.495 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:25.495+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:25.594 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:25.594+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:25.596 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:25.596 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:25.596 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:25.596 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:25.597 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:25.604 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:25.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:25.702 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:25.702+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:25.792 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.37576251199999, 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:25.797 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:25.797 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:25.797 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:25.798 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:25.798+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:25.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:25.899 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:25.899+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:25.932 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:25.932 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:25.932 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:25.932 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:25.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:26.000 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:25.999+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:26.068 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:26.068 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:26.068 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:26.068 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:26.068 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:26.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:26.101 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:26.101+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:26.202 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:26.201+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:26.246 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:26.246 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:26.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:26.246 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:26.247 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:26.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T07:59:26.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:26.340 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:26.339+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:26.404 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:26.403+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:26.427 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:26.428 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:26.428 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:26.428 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:26.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:26.504 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:26.504+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:26.605 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:26.605 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:26.605 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:26.605+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:26.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:26.706 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:26.706+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:26.809 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:26.808+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 125.84276103800001, 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:59:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:59:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 125.178980037, 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:59:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 125.175699408, 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:59:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:59:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.870922792, 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:59:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:59:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.870079695, 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:59:26.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:26.848 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.847544189, 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.84555177, 2023-12-06T07:59:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:59:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.845019505, 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:59:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.828094596, 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:59:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:59:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:59:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 124.827257962, 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.802810694, 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.79690832999999, 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:26.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:59:26.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:26.876 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.792202334, 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.78990045899999, 2023-12-06T07:59:26.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:59:26.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.78940213600001, 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.788948225, 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:59:26.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:59:26.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 123.787725825, 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:26.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:59:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:59:26.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:59:26.896 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 118.142559844, 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:26.900 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 118.13512142899999, 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 118.125048057, 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:59:26.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:59:26.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:59:26.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:26.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:26.908 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:26.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:26.910 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:26.909+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:27.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:59:27.050 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:27.050+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:27.077 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:27.077 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:27.077 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:27.077 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:27.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:27.112 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:27.112+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:27.213 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:27.212+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:27.215 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:27.215 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:27.215 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:27.215 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:27.216 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:27.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:27.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:27.237 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:27.314 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:27.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:27.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:27.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:27.351 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:27.352 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:27.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:27.352 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:27.352 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:27.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:27.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:27.405 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:27.415 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:27.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:27.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:27.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:27.488 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:27.488 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:27.488 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:27.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:27.516 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:27.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:27.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:27.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:27.626 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:27.626 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:27.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:27.662 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:27.661+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:27.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:27.663 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:27.718 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:27.718+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:27.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:27.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:27.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:27.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:27.843 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:27.842+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:27.845 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:27.845 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:27.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:27.920 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:27.920+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:27.997 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T07:59:28.013 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.013 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:28.022 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:28.022+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:28.123 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:28.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:28.191 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.191 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:28.224 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:28.224+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:28.325 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:28.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:28.371 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.371 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:28.426 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:28.426+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:28.452 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:59:28.452 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:59:28.452 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2023-12-06T07:59:28.452 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T07:59:28.452 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T07:59:28.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2023-12-06T07:59:28.512 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.513 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:28.527 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:28.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:28.590 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T07:59:28.590 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:28.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:28.656 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.656 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.657 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:28.656+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:28.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:28.752 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:28.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:28.829 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:28.829+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:28.841 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:28.841 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:28.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:28.930 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:28.929+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:29.029 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.029 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.031 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:29.030+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:29.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:29.132 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:29.132+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:29.172 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.172 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:29.233 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:29.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:29.279 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:29.334 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:29.334+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:29.353 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.353 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:29.436 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:29.435+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:29.536 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:29.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:29.553 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.553 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:29.646 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:29.646+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:29.733 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.734 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.738 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:29.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:29.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:29.839 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:29.839+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:29.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:59:29.935 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:29.935 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:29.940 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:29.940+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:29.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:30.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T07:59:30.059 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:30.059+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:30.076 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:30.076 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:30.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:30.142 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:30.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:30.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:30.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:30.215 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:30.215 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:30.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:30.272 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:30.271+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:30.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:30.302 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:30.302 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.386 95130== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:30.346 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:30.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:30.361 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:30.361 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:30.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:30.385 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.468 95130== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:30.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:30.387 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:30.446 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:30.446+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:30.473 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:30.473 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:30.522 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:30.522 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:30.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:30.547 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:30.547+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:30.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:30.568 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:30.598 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.682 95130== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:30.648 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:30.647+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:30.658 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:30.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:30.707 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:30.708 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:30.708 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:30.708 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:30.708 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:30.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:30.760 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:30.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:30.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:30.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:30.849 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:30.849+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:30.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:30.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:30.951 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:30.951+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 129.968765112, 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T07:59:30.955 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T07:59:30.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 129.30498411100001, 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T07:59:30.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 129.30170348199999, 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T07:59:30.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.966 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.996926866, 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T07:59:30.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T07:59:30.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.972 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.99608376899999, 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T07:59:30.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T07:59:30.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T07:59:30.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.973548263, 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:30.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T07:59:30.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T07:59:30.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T07:59:30.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.97155584399999, 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T07:59:30.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T07:59:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T07:59:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T07:59:30.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.97102357899999, 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.95409867000001, 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T07:59:30.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.993 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.95326203600001, 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:30.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T07:59:30.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:30.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:30.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.928814768, 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.000 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:31.004 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.922912404, 2023-12-06T07:59:31.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T07:59:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.918206408, 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.010 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T07:59:31.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T07:59:31.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.915904533, 2023-12-06T07:59:31.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T07:59:31.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T07:59:31.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T07:59:31.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.91540621, 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.019 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T07:59:31.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T07:59:31.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T07:59:31.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.91495229900001, 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 127.913729899, 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.028 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T07:59:31.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T07:59:31.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T07:59:31.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 122.268563918, 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T07:59:31.032 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T07:59:31.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T07:59:31.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 122.261125503, 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.036 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T07:59:31.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T07:59:31.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:31.040 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 122.25105213099999, 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T07:59:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T07:59:31.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T07:59:31.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.045 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:31.045 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:31.045 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:31.090 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:31.089+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:31.158 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:31.157+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:31.181 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:31.181 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:31.181 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:31.182 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:31.254 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:31.253+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:31.354 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:31.354+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:31.370 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:31.371 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:31.458 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:31.458+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:31.549 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T07:59:31.550 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:31.560 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:31.559+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:31.633 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T07:59:31.663 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:31.662+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:31.698 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T07:59:31.699 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:31.793 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:31.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:31.866 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:31.865+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:31.877 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:31.877 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T07:59:31.878 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T07:59:31.878 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T07:59:31.878 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:31.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:31.967 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:31.966+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:32.060 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 111.484276428, 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T07:59:32.061 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T07:59:32.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T07:59:32.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:32.064 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:32.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:32.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T07:59:32.065 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T07:59:32.065 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T07:59:32.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:32.072 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:32.070+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:32.172 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:32.172+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:32.207 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:32.207 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:32.208 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:32.208 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:32.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:32.274 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:32.274+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:32.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:32.348 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:32.348 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:32.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:32.403 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:32.402+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:32.476 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:32.476+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:32.491 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:32.491 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:32.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:32.491 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T07:59:32.491 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:32.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:32.584 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:32.583+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:32.671 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T07:59:32.672 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:32.677 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:32.677+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:32.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:32.778 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:32.777+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:32.872 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:32.873 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T07:59:32.873 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T07:59:32.873 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T07:59:32.873 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:32.879 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:32.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:32.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:32.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T07:59:32.979 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T07:59:32.979 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:32.978+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:32.993 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T07:59:32.993 INFO:tasks.mon_recovery:stopping all monitors 2023-12-06T07:59:32.993 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T07:59:32.993 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:59:33.009 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T07:59:32.997+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 95128) UID: 0 2023-12-06T07:59:33.010 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T07:59:32.998+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T07:59:33.080 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:33.079+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:33.123 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:33.123 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T07:59:33.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T07:59:33.123 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 132.990594767, 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T07:59:33.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T07:59:33.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T07:59:33.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T07:59:33.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.132 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.51948131399999, 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:33.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T07:59:33.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T07:59:33.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.138 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.289288935, 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.139 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T07:59:33.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T07:59:33.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T07:59:33.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.288581386, 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.28794439199999, 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T07:59:33.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T07:59:33.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T07:59:33.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.23087919700001, 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.155 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T07:59:33.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.22938842600001, 2023-12-06T07:59:33.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T07:59:33.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T07:59:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.19703876400001, 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T07:59:33.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T07:59:33.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.170 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.19040798899999, 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T07:59:33.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.189757308, 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T07:59:33.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.18628461700001, 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.181 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T07:59:33.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T07:59:33.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T07:59:33.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.18514589500001, 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.18419819799999, 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T07:59:33.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T07:59:33.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.155834204, 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.197 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T07:59:33.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T07:59:33.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.12762468700001, 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T07:59:33.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T07:59:33.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T07:59:33.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T07:59:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:33.207 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.517426753, 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.516781913, 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:33.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T07:59:33.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T07:59:33.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.217 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.49631515900001, 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T07:59:33.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T07:59:33.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.495566243, 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.223 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.494924174, 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T07:59:33.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T07:59:33.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T07:59:33.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T07:59:33.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T07:59:33.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:33.233 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:33.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:33.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T07:59:33.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T07:59:33.236 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:33.234+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:33.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T07:59:33.265 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T07:59:33.335 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:33.334+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:33.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T07:59:33.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T07:59:33.412 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:33.412 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:33.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:33.463 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T07:59:33.463 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T07:59:33.464 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:33.462+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:33.540 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:33.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:33.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T07:59:33.555 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T07:59:33.591 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:33.591 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:33.597 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:23.681 95130== 2023-12-06T07:59:33.597 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:23.681 95130== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:59:33.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:33.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T07:59:33.600 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T07:59:33.600 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T07:59:33.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T07:59:33.606 INFO:tasks.ceph.mon.b.smithi107.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:59:33.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T07:59:33.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T07:59:33.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T07:59:33.654 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:33.654+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T07:59:33.729 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T07:59:33.742 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:33.740+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:33.779 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:33.779 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:33.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:33.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T07:59:33.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T07:59:33.843 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:33.842+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:33.943 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:33.943+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:33.954 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:33.954 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:33.963 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:34.045 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:34.045+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:34.095 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.095 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:34.149 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:34.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:34.236 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.236 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:34.247 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:34.246+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:34.348 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:34.347+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:34.375 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.375 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.377 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:59:34.377 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T07:59:34.377 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 44, 2023-12-06T07:59:34.377 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T07:59:31.508176+0000", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T07:59:34.378 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T07:59:34.379 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T07:59:34.380 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:59:34.381 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:59:34.382 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:59:34.383 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:59:34.384 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T07:59:34.386 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T07:59:34.387 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T07:59:34.388 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T07:59:34.389 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T07:59:34.390 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3.9800000190734863, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3.9800000190734863, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 3.5, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 3.5, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.392 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T07:59:34.393 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.396 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 30, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T07:59:34.400 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:34.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.405 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.406 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.407 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.409 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.412 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T07:59:34.413 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.414 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.416 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.418 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.419 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.420 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T07:59:34.422 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 24, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.423 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.424 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T07:59:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.428 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.429 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.430 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.432 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2023-12-06T07:59:34.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.5", 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: 6 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T07:59:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 1 2023-12-06T07:59:34.435 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T07:59:34.436 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T07:59:34.437 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T07:59:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T07:59:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T07:59:34.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T07:59:34.438 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T07:59:34.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:34.442 INFO:tasks.thrashosds.thrasher:j is {'epoch': 44, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T07:59:31.508176+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.9800000190734863, 'score_stable': 3.9800000190734863, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 3.5, 'raw_score_stable': 3.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [1, 6]}, {'pgid': '2.6', 'osds': [7, 2]}], 'pg_upmap_items': [{'pgid': '2.6', 'mappings': [{'from': 0, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T07:59:34.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T07:59:34.486 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:34.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:34.550 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:34.549+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:34.606 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.607 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:34.657 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:34.657+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:34.751 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:34.750+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:34.783 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.783 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:34.852 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:34.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:34.920 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:34.920 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:34.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:34.953 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:34.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:35.001 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T07:59:35.001 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T07:59:35.013 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2023-12-06T07:59:33.823642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":220184,"kb_used_data":4024,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754536,"statfs":{"total":773094113280,"available":772868644864,"internally_reserved":0,"allocated":4120576,"data_stored":2030566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001369"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:25.394935+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:59:25.394935+0000","last_peered":"2023-12-06T07:59:25.394935+0000","last_clean":"2023-12-06T07:59:25.394935+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:59:25.394935+0000","last_undegraded":"2023-12-06T07:59:25.394935+0000","last_fullsized":"2023-12-06T07:59:25.394935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:24.567638+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T07:59:24.567638+0000","last_peered":"2023-12-06T07:59:24.567638+0000","last_clean":"2023-12-06T07:59:24.567638+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T07:59:24.567638+0000","last_undegraded":"2023-12-06T07:59:24.567638+0000","last_fullsized":"2023-12-06T07:59:24.567638+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:24.587883+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T07:59:24.587883+0000","last_peered":"2023-12-06T07:59:24.587883+0000","last_clean":"2023-12-06T07:59:24.587883+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T07:59:24.587883+0000","last_undegraded":"2023-12-06T07:59:24.587883+0000","last_fullsized":"2023-12-06T07:59:24.587883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:24.160524+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:59:24.160524+0000","last_peered":"2023-12-06T07:59:24.160524+0000","last_clean":"2023-12-06T07:59:24.160524+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:59:24.160524+0000","last_undegraded":"2023-12-06T07:59:24.160524+0000","last_fullsized":"2023-12-06T07:59:24.160524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:25.391495+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:59:25.391495+0000","last_peered":"2023-12-06T07:59:25.391495+0000","last_clean":"2023-12-06T07:59:25.391495+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:59:25.391495+0000","last_undegraded":"2023-12-06T07:59:25.391495+0000","last_fullsized":"2023-12-06T07:59:25.391495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:24.562783+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:59:24.562783+0000","last_peered":"2023-12-06T07:59:24.562783+0000","last_clean":"2023-12-06T07:59:24.562783+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:59:24.562783+0000","last_undegraded":"2023-12-06T07:59:24.562783+0000","last_fullsized":"2023-12-06T07:59:24.562783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-06T07:59:29.558955+0000","last_change":"2023-12-06T07:57:30.863236+0000","last_active":"2023-12-06T07:59:29.558955+0000","last_peered":"2023-12-06T07:59:29.558955+0000","last_clean":"2023-12-06T07:59:29.558955+0000","last_became_active":"2023-12-06T07:57:30.832265+0000","last_became_peered":"2023-12-06T07:57:30.832265+0000","last_unstale":"2023-12-06T07:59:29.558955+0000","last_undegraded":"2023-12-06T07:59:29.558955+0000","last_fullsized":"2023-12-06T07:59:29.558955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:45:53.821733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:25.393198+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:59:25.393198+0000","last_peered":"2023-12-06T07:59:25.393198+0000","last_clean":"2023-12-06T07:59:25.393198+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:59:25.393198+0000","last_undegraded":"2023-12-06T07:59:25.393198+0000","last_fullsized":"2023-12-06T07:59:25.393198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":140,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-12-06T07:59:24.589771+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:59:24.589771+0000","last_peered":"2023-12-06T07:59:24.589771+0000","last_clean":"2023-12-06T07:59:24.589771+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:59:24.589771+0000","last_undegraded":"2023-12-06T07:59:24.589771+0000","last_fullsized":"2023-12-06T07:59:24.589771+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476765,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":417792,"data_stored":155463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":9.5679999999999996,"5min":9.5860000000000003,"15min":9.5890000000000004},"min":{"1min":5.202,"5min":5.202,"15min":5.202},"max":{"1min":27.638999999999999,"5min":27.638999999999999,"15min":27.638999999999999},"last":5.3799999999999999},{"interface":"front","average":{"1min":9.7639999999999993,"5min":8.8399999999999999,"15min":8.6859999999999999},"min":{"1min":5.0309999999999997,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":24.085000000000001,"5min":24.085000000000001,"15min":24.085000000000001},"last":8.5099999999999998}]},{"osd":1,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":10.199,"5min":12.532999999999999,"15min":12.922000000000001},"min":{"1min":6.3970000000000002,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":21.231999999999999,"5min":29.213000000000001,"15min":29.213000000000001},"last":7.4720000000000004},{"interface":"front","average":{"1min":13.334,"5min":14.449999999999999,"15min":14.635999999999999},"min":{"1min":9.0530000000000008,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":23.311,"5min":39.337000000000003,"15min":39.337000000000003},"last":14.009}]},{"osd":2,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":11.083,"5min":12.324999999999999,"15min":12.532},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":17.152000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":10.477},{"interface":"front","average":{"1min":14.012,"5min":14.178000000000001,"15min":14.206},"min":{"1min":7.4470000000000001,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":26.388000000000002,"5min":26.388000000000002,"15min":26.388000000000002},"last":9.2639999999999993}]},{"osd":4,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":15.282,"5min":13.589,"15min":13.307},"min":{"1min":10.794,"5min":8.6259999999999994,"15min":8.6259999999999994},"max":{"1min":20.471,"5min":23.986000000000001,"15min":23.986000000000001},"last":14.833},{"interface":"front","average":{"1min":16.187000000000001,"5min":16.135999999999999,"15min":16.128},"min":{"1min":8.0500000000000007,"5min":7.806,"15min":7.806},"max":{"1min":54.225999999999999,"5min":54.225999999999999,"15min":54.225999999999999},"last":12.144}]},{"osd":5,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":16.652999999999999,"5min":17.109000000000002,"15min":17.184999999999999},"min":{"1min":9.6880000000000006,"5min":8.8580000000000005,"15min":8.8580000000000005},"max":{"1min":29.523,"5min":29.523,"15min":29.523},"last":19.202999999999999},{"interface":"front","average":{"1min":15.603,"5min":17.038,"15min":17.277000000000001},"min":{"1min":8.3870000000000005,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":21.338999999999999,"5min":38.670000000000002,"15min":38.670000000000002},"last":17.838999999999999}]},{"osd":6,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":12.997999999999999,"5min":14.481,"15min":14.728},"min":{"1min":6.3680000000000003,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":27.890000000000001,"5min":27.890000000000001,"15min":27.890000000000001},"last":11.182},{"interface":"front","average":{"1min":19.378,"5min":18.928000000000001,"15min":18.853000000000002},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":44.255000000000003,"5min":44.255000000000003,"15min":44.255000000000003},"last":13.282}]},{"osd":7,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":17.896000000000001,"5min":19.148,"15min":19.356999999999999},"min":{"1min":12.214,"5min":12.214,"15min":12.214},"max":{"1min":23.091000000000001,"5min":40.832000000000001,"15min":40.832000000000001},"last":15.429},{"interface":"front","average":{"1min":17.309000000000001,"5min":18.459,"15min":18.651},"min":{"1min":10.505000000000001,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":29.32,"5min":39.331000000000003,"15min":39.331000000000003},"last":16.053000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476766,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":151054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":8.9700000000000006,"5min":10.191000000000001,"15min":10.395},"min":{"1min":4.5359999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":17.594999999999999,"5min":22.759,"15min":22.759},"last":11.627000000000001},{"interface":"front","average":{"1min":10.936999999999999,"5min":10.676,"15min":10.632},"min":{"1min":5.9180000000000001,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":30.768999999999998,"5min":34.981000000000002,"15min":34.981000000000002},"last":14.109999999999999}]},{"osd":1,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":13.324999999999999,"5min":20.66,"15min":21.882000000000001},"min":{"1min":7.6429999999999998,"5min":6.7800000000000002,"15min":6.7800000000000002},"max":{"1min":27.763000000000002,"5min":165.68600000000001,"15min":165.68600000000001},"last":9.9580000000000002},{"interface":"front","average":{"1min":11.452999999999999,"5min":20.914999999999999,"15min":22.492000000000001},"min":{"1min":6.3760000000000003,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":27.297999999999998,"5min":153.68199999999999,"15min":153.68199999999999},"last":10.561999999999999}]},{"osd":3,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":8.9290000000000003,"5min":10.803000000000001,"15min":11.115},"min":{"1min":5.4299999999999997,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":14.581,"5min":35.935000000000002,"15min":35.935000000000002},"last":8.0589999999999993},{"interface":"front","average":{"1min":16.529,"5min":15.874000000000001,"15min":15.765000000000001},"min":{"1min":9.4629999999999992,"5min":8.6560000000000006,"15min":8.6560000000000006},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":15.551}]},{"osd":4,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":11.83,"5min":15.986000000000001,"15min":16.678999999999998},"min":{"1min":7.0830000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":17.033000000000001,"5min":50.137999999999998,"15min":50.137999999999998},"last":12.393000000000001},{"interface":"front","average":{"1min":17.085000000000001,"5min":16.704999999999998,"15min":16.641999999999999},"min":{"1min":9.9350000000000005,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":31.765000000000001,"5min":46.325000000000003,"15min":46.325000000000003},"last":18.170000000000002}]},{"osd":5,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":19.056000000000001,"5min":20.315999999999999,"15min":20.526},"min":{"1min":10.106,"5min":10.106,"15min":10.106},"max":{"1min":32.866,"5min":50.320999999999998,"15min":50.320999999999998},"last":19.34},{"interface":"front","average":{"1min":14.555,"5min":17.138999999999999,"15min":17.568999999999999},"min":{"1min":7.9370000000000003,"5min":7.9370000000000003,"15min":7.9370000000000003},"max":{"1min":25.809999999999999,"5min":44.323,"15min":44.323},"last":9.0449999999999999}]},{"osd":6,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":19.661999999999999,"15min":20.277999999999999},"min":{"1min":9.2330000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":48.231000000000002,"5min":104.834,"15min":104.834},"last":13.051},{"interface":"front","average":{"1min":22.042999999999999,"5min":21.042999999999999,"15min":20.876000000000001},"min":{"1min":12.053000000000001,"5min":12.053000000000001,"15min":12.053000000000001},"max":{"1min":49.146999999999998,"5min":49.146999999999998,"15min":49.146999999999998},"last":16.756}]},{"osd":7,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":17.242999999999999,"5min":19.402999999999999,"15min":19.763000000000002},"min":{"1min":10.637,"5min":7.8440000000000003,"15min":7.8440000000000003},"max":{"1min":27.006,"5min":42.915999999999997,"15min":42.915999999999997},"last":14.827},{"interface":"front","average":{"1min":16.879999999999999,"5min":18.922000000000001,"15min":19.262},"min":{"1min":9.8239999999999998,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":26.298999999999999,"5min":42.506,"15min":42.506},"last":17.381}]}]},{"osd":1,"up_from":16,"seq":68719476766,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":553303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":9.718,"5min":12.887,"15min":13.414999999999999},"min":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":5.0890000000000004},"max":{"1min":19.373999999999999,"5min":80.640000000000001,"15min":80.640000000000001},"last":6.1600000000000001},{"interface":"front","average":{"1min":8.9740000000000002,"5min":11.124000000000001,"15min":11.481999999999999},"min":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"max":{"1min":18.952000000000002,"5min":25.390000000000001,"15min":25.390000000000001},"last":9.7989999999999995}]},{"osd":2,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.468,"5min":19.667000000000002,"15min":20.699999999999999},"min":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.533000000000001,"5min":100.703,"15min":100.703},"last":10.548999999999999},{"interface":"front","average":{"1min":13.167999999999999,"5min":18.32,"15min":19.178999999999998},"min":{"1min":6.532,"5min":6.532,"15min":6.532},"max":{"1min":23.036999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":6.8840000000000003}]},{"osd":3,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":14.539,"5min":17.850999999999999,"15min":18.402999999999999},"min":{"1min":8.0250000000000004,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":29.652999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":18.204999999999998},{"interface":"front","average":{"1min":9.5229999999999997,"5min":13.081,"15min":13.673999999999999},"min":{"1min":6.3390000000000004,"5min":6.3390000000000004,"15min":6.3390000000000004},"max":{"1min":15.93,"5min":46.613,"15min":46.613},"last":8.9830000000000005}]},{"osd":4,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":15.394,"5min":18.117999999999999,"15min":18.571999999999999},"min":{"1min":8.4450000000000003,"5min":8.4450000000000003,"15min":8.4450000000000003},"max":{"1min":23.760999999999999,"5min":81.838999999999999,"15min":81.838999999999999},"last":11.657},{"interface":"front","average":{"1min":16.411999999999999,"5min":20.725999999999999,"15min":21.445},"min":{"1min":10.313000000000001,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":27.658000000000001,"5min":66.515000000000001,"15min":66.515000000000001},"last":23.018000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":12.662000000000001,"5min":15.837999999999999,"15min":16.367999999999999},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":28.388000000000002,"5min":56.235999999999997,"15min":56.235999999999997},"last":22.239999999999998},{"interface":"front","average":{"1min":17.988,"5min":19.594000000000001,"15min":19.861999999999998},"min":{"1min":9.766,"5min":9.766,"15min":9.766},"max":{"1min":27.745000000000001,"5min":82.381,"15min":82.381},"last":12.452}]},{"osd":6,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":17.834,"15min":18.564},"min":{"1min":9.1959999999999997,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":29.748000000000001,"5min":58.619,"15min":58.619},"last":19.523},{"interface":"front","average":{"1min":18.754000000000001,"5min":20.972000000000001,"15min":21.341999999999999},"min":{"1min":9.6270000000000007,"5min":9.6229999999999993,"15min":9.6229999999999993},"max":{"1min":33.616999999999997,"5min":83.632000000000005,"15min":83.632000000000005},"last":7.8529999999999998}]},{"osd":7,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":18.992000000000001,"5min":22.225999999999999,"15min":22.765000000000001},"min":{"1min":13.016,"5min":13.016,"15min":13.016},"max":{"1min":32.531999999999996,"5min":62.006999999999998,"15min":62.006999999999998},"last":24.189},{"interface":"front","average":{"1min":15.484999999999999,"5min":19.895,"15min":20.629999999999999},"min":{"1min":10.445,"5min":9.9740000000000002,"15min":9.9740000000000002},"max":{"1min":25.457000000000001,"5min":60.698999999999998,"15min":60.698999999999998},"last":21.649000000000001}]}]},{"osd":0,"up_from":16,"seq":68719476767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":151054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":9.1500000000000004,"5min":18.41,"15min":19.952999999999999},"min":{"1min":5.6470000000000002,"5min":5.6470000000000002,"15min":5.6470000000000002},"max":{"1min":19.452000000000002,"5min":252.88399999999999,"15min":252.88399999999999},"last":6.8460000000000001},{"interface":"front","average":{"1min":8.9000000000000004,"5min":19.289000000000001,"15min":21.021000000000001},"min":{"1min":5.9649999999999999,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":17.972000000000001,"5min":274.91500000000002,"15min":274.91500000000002},"last":13.074999999999999}]},{"osd":2,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":11.615,"5min":16.530999999999999,"15min":17.350999999999999},"min":{"1min":6.2649999999999997,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":20.821999999999999,"5min":82.866,"15min":82.866},"last":9.6549999999999994},{"interface":"front","average":{"1min":13.494999999999999,"5min":15.962999999999999,"15min":16.373999999999999},"min":{"1min":6.7930000000000001,"5min":6.7930000000000001,"15min":6.7930000000000001},"max":{"1min":20.969999999999999,"5min":85.991,"15min":85.991},"last":20.097000000000001}]},{"osd":3,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":10.926,"5min":14.554,"15min":15.157999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":28.27,"5min":76.593000000000004,"15min":76.593000000000004},"last":28.27},{"interface":"front","average":{"1min":13.904,"5min":19.632000000000001,"15min":20.585999999999999},"min":{"1min":6.9960000000000004,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":30.224,"5min":85.162000000000006,"15min":85.162000000000006},"last":30.224}]},{"osd":4,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":14.275,"5min":16.917000000000002,"15min":17.356999999999999},"min":{"1min":9.1389999999999993,"5min":7.516,"15min":7.516},"max":{"1min":22.143000000000001,"5min":65.349000000000004,"15min":65.349000000000004},"last":11.291},{"interface":"front","average":{"1min":10.933999999999999,"5min":13.465999999999999,"15min":13.888},"min":{"1min":6.5780000000000003,"5min":6.5780000000000003,"15min":6.5780000000000003},"max":{"1min":18.934000000000001,"5min":70.085999999999999,"15min":70.085999999999999},"last":14.872}]},{"osd":5,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":18.460000000000001,"5min":26.390000000000001,"15min":27.712},"min":{"1min":9.8559999999999999,"5min":9.8559999999999999,"15min":9.8559999999999999},"max":{"1min":46.792000000000002,"5min":144.60499999999999,"15min":144.60499999999999},"last":18.969999999999999},{"interface":"front","average":{"1min":17.689,"5min":24.901,"15min":26.103000000000002},"min":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":45.198,"5min":142.25,"15min":142.25},"last":15.941000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":13.087999999999999,"5min":17.364999999999998,"15min":18.077999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":17.132000000000001,"5min":75.471999999999994,"15min":75.471999999999994},"last":7.6799999999999997},{"interface":"front","average":{"1min":16.850000000000001,"5min":22.954000000000001,"15min":23.971},"min":{"1min":9.4589999999999996,"5min":9.4589999999999996,"15min":9.4589999999999996},"max":{"1min":25.654,"5min":84.617000000000004,"15min":84.617000000000004},"last":16.721}]},{"osd":7,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":17.565000000000001,"5min":21.297000000000001,"15min":21.919},"min":{"1min":9.6890000000000001,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":28.094000000000001,"5min":84.031000000000006,"15min":84.031000000000006},"last":17.494},{"interface":"front","average":{"1min":14.311,"5min":18.312000000000001,"15min":18.978999999999999},"min":{"1min":9.6310000000000002,"5min":9.6310000000000002,"15min":9.6310000000000002},"max":{"1min":23.643000000000001,"5min":77.974000000000004,"15min":77.974000000000004},"last":10.382999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509471,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27428,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":417792,"data_stored":155463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":7.1479999999999997,"5min":9.2040000000000006,"15min":9.5459999999999994},"min":{"1min":4.234,"5min":4.234,"15min":4.234},"max":{"1min":16.420000000000002,"5min":23.677,"15min":23.677},"last":5.5419999999999998},{"interface":"front","average":{"1min":9.7490000000000006,"5min":10.183,"15min":10.255000000000001},"min":{"1min":5.0960000000000001,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":18.917999999999999,"5min":22.443999999999999,"15min":22.443999999999999},"last":6.3570000000000002}]},{"osd":1,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":9.8130000000000006,"5min":11.587,"15min":11.882999999999999},"min":{"1min":5.0090000000000003,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":18.620999999999999,"5min":35.524000000000001,"15min":35.524000000000001},"last":6.9699999999999998},{"interface":"front","average":{"1min":13.038,"5min":13.85,"15min":13.984999999999999},"min":{"1min":7.2859999999999996,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":19.516999999999999,"5min":31.027000000000001,"15min":31.027000000000001},"last":8.2579999999999991}]},{"osd":2,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.407999999999999,"5min":12.295,"15min":12.109},"min":{"1min":9.0310000000000006,"5min":8.1159999999999997,"15min":8.1159999999999997},"max":{"1min":22.754000000000001,"5min":22.754000000000001,"15min":22.754000000000001},"last":12.013999999999999},{"interface":"front","average":{"1min":15.56,"5min":14.811999999999999,"15min":14.688000000000001},"min":{"1min":10.954000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":32.915999999999997,"5min":32.915999999999997,"15min":32.915999999999997},"last":11.574999999999999}]},{"osd":3,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":10.792,"5min":11.346,"15min":11.438000000000001},"min":{"1min":6.9939999999999998,"5min":6.843,"15min":6.843},"max":{"1min":17.343,"5min":25.640999999999998,"15min":25.640999999999998},"last":16.795000000000002},{"interface":"front","average":{"1min":15.763,"5min":14.404,"15min":14.178000000000001},"min":{"1min":9.282,"5min":8.6170000000000009,"15min":8.6170000000000009},"max":{"1min":25.077999999999999,"5min":25.077999999999999,"15min":25.077999999999999},"last":15.425000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":16.332999999999998,"5min":16.945,"15min":17.047000000000001},"min":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":36.713000000000001,"5min":50.417999999999999,"15min":50.417999999999999},"last":53.899999999999999},{"interface":"front","average":{"1min":18.265999999999998,"5min":17.788,"15min":17.707999999999998},"min":{"1min":12.552,"5min":9.9399999999999995,"15min":9.9399999999999995},"max":{"1min":26.803999999999998,"5min":48.837000000000003,"15min":48.837000000000003},"last":55.914000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":17.388000000000002,"5min":21.440000000000001,"15min":22.116},"min":{"1min":8.7899999999999991,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":40.418999999999997,"5min":93.073999999999998,"15min":93.073999999999998},"last":12.949999999999999},{"interface":"front","average":{"1min":12.763999999999999,"5min":17.523,"15min":18.315999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":21.292000000000002,"5min":91.075999999999993,"15min":91.075999999999993},"last":13.752000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":16.18,"15min":16.305},"min":{"1min":9.8279999999999994,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":33.988999999999997,"5min":33.988999999999997,"15min":33.988999999999997},"last":17.736000000000001},{"interface":"front","average":{"1min":19.609000000000002,"5min":17.306000000000001,"15min":16.922000000000001},"min":{"1min":14.167999999999999,"5min":12.539,"15min":12.539},"max":{"1min":32.014000000000003,"5min":32.014000000000003,"15min":32.014000000000003},"last":18.577999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":417792,"data_stored":155463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":8.8780000000000001,"5min":10.023999999999999,"15min":10.215},"min":{"1min":4.2779999999999996,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":15.942,"5min":70.965000000000003,"15min":70.965000000000003},"last":11.815},{"interface":"front","average":{"1min":8.6679999999999993,"5min":11.603999999999999,"15min":12.093},"min":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":15.984999999999999,"5min":73.582999999999998,"15min":73.582999999999998},"last":26.015999999999998}]},{"osd":1,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":13.266,"5min":18.300000000000001,"15min":19.138999999999999},"min":{"1min":7.3079999999999998,"5min":4.383,"15min":4.383},"max":{"1min":39.470999999999997,"5min":217.595,"15min":217.595},"last":6.7800000000000002},{"interface":"front","average":{"1min":12.566000000000001,"5min":21.143000000000001,"15min":22.573},"min":{"1min":5.9859999999999998,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":37.878,"5min":284.96499999999997,"15min":284.96499999999997},"last":7.7709999999999999}]},{"osd":2,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":12.776999999999999,"5min":13.661,"15min":13.808},"min":{"1min":7.3120000000000003,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":19.815000000000001,"5min":46.024999999999999,"15min":46.024999999999999},"last":10.271000000000001},{"interface":"front","average":{"1min":12.541,"5min":16.094999999999999,"15min":16.687000000000001},"min":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"max":{"1min":19.689,"5min":46.966999999999999,"15min":46.966999999999999},"last":11.308}]},{"osd":3,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":14.635999999999999,"5min":15.603999999999999,"15min":15.765000000000001},"min":{"1min":9.3460000000000001,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":25.395,"5min":51.031999999999996,"15min":51.031999999999996},"last":30.462},{"interface":"front","average":{"1min":13.731,"5min":22.231000000000002,"15min":23.646999999999998},"min":{"1min":7.9880000000000004,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":21.751000000000001,"5min":189.84999999999999,"15min":189.84999999999999},"last":13.363}]},{"osd":4,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":15.287000000000001,"5min":15.763999999999999,"15min":15.843999999999999},"min":{"1min":10.183,"5min":10.183,"15min":10.183},"max":{"1min":20.337,"5min":25.013000000000002,"15min":25.013000000000002},"last":55.661999999999999},{"interface":"front","average":{"1min":15.156000000000001,"5min":14.734999999999999,"15min":14.664999999999999},"min":{"1min":10.518000000000001,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":22.789000000000001,"5min":26.831,"15min":26.831},"last":36.552}]},{"osd":5,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":19.088000000000001,"5min":18.956,"15min":18.934000000000001},"min":{"1min":10.811,"5min":10.811,"15min":10.811},"max":{"1min":43.795999999999999,"5min":50.945,"15min":50.945},"last":32.976999999999997},{"interface":"front","average":{"1min":14.279,"5min":14.353999999999999,"15min":14.366},"min":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"max":{"1min":45.25,"5min":51.948,"15min":51.948},"last":29.141999999999999}]},{"osd":7,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":16.399999999999999,"5min":15.888,"15min":15.803000000000001},"min":{"1min":9.1069999999999993,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":24.576000000000001,"5min":31.353999999999999,"15min":31.353999999999999},"last":28.512},{"interface":"front","average":{"1min":17.803999999999998,"5min":19.163,"15min":19.388999999999999},"min":{"1min":12.43,"5min":12.43,"15min":12.43},"max":{"1min":30.562000000000001,"5min":30.562000000000001,"15min":30.562000000000001},"last":14.606}]}]},{"osd":5,"up_from":14,"seq":60129542174,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":553303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":8.0700000000000003,"5min":8.923,"15min":9.0649999999999995},"min":{"1min":5.1520000000000001,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":15.169,"5min":23.09,"15min":23.09},"last":6.6500000000000004},{"interface":"front","average":{"1min":8.5649999999999995,"5min":9.4199999999999999,"15min":9.5619999999999994},"min":{"1min":4.7930000000000001,"5min":4.7930000000000001,"15min":4.7930000000000001},"max":{"1min":19.276,"5min":20.728000000000002,"15min":20.728000000000002},"last":8.1059999999999999}]},{"osd":1,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":11.742000000000001,"5min":13.407999999999999,"15min":13.686},"min":{"1min":4.8979999999999997,"5min":4.8979999999999997,"15min":4.8979999999999997},"max":{"1min":24.928000000000001,"5min":67.347999999999999,"15min":67.347999999999999},"last":14.630000000000001},{"interface":"front","average":{"1min":13.577,"5min":12.922000000000001,"15min":12.813000000000001},"min":{"1min":6.2640000000000002,"5min":5.8090000000000002,"15min":5.8090000000000002},"max":{"1min":23.263999999999999,"5min":39.052,"15min":39.052},"last":7.4219999999999997}]},{"osd":2,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.504,"5min":12.599,"15min":12.281000000000001},"min":{"1min":7.1639999999999997,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":31.873000000000001,"5min":31.873000000000001,"15min":31.873000000000001},"last":14.912000000000001},{"interface":"front","average":{"1min":14.506,"5min":15.742000000000001,"15min":15.948},"min":{"1min":6.8209999999999997,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":30.876000000000001,"5min":34.585000000000001,"15min":34.585000000000001},"last":8.5109999999999992}]},{"osd":3,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":15.291,"5min":15.486000000000001,"15min":15.518000000000001},"min":{"1min":9.5589999999999993,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":28.518000000000001,"5min":30.355,"15min":30.355},"last":10.696},{"interface":"front","average":{"1min":15.138999999999999,"5min":15.185,"15min":15.193},"min":{"1min":7.9359999999999999,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":31.792000000000002,"5min":31.792000000000002,"15min":31.792000000000002},"last":13.236000000000001}]},{"osd":4,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":13.646000000000001,"5min":26.282,"15min":28.388000000000002},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":18.556999999999999,"5min":334.86500000000001,"15min":334.86500000000001},"last":9.2370000000000001},{"interface":"front","average":{"1min":14.972,"5min":26.417000000000002,"15min":28.324999999999999},"min":{"1min":10.587,"5min":7.431,"15min":7.431},"max":{"1min":23.646999999999998,"5min":331.57900000000001,"15min":331.57900000000001},"last":11.855}]},{"osd":6,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":15.94,"5min":19.248000000000001,"15min":19.798999999999999},"min":{"1min":9.6400000000000006,"5min":9.6400000000000006,"15min":9.6400000000000006},"max":{"1min":27.963999999999999,"5min":46.034999999999997,"15min":46.034999999999997},"last":16.135999999999999},{"interface":"front","average":{"1min":16.256,"5min":18.359999999999999,"15min":18.710000000000001},"min":{"1min":9.6709999999999994,"5min":9.6709999999999994,"15min":9.6709999999999994},"max":{"1min":25.742000000000001,"5min":35.680999999999997,"15min":35.680999999999997},"last":11.281000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":27.978999999999999,"15min":30.210000000000001},"min":{"1min":9.1329999999999991,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":26.213000000000001,"5min":332.322,"15min":332.322},"last":16.966000000000001},{"interface":"front","average":{"1min":16.748999999999999,"5min":30.989000000000001,"15min":33.363},"min":{"1min":11.039,"5min":11.039,"15min":11.039},"max":{"1min":21.757000000000001,"5min":325.93599999999998,"15min":325.93599999999998},"last":17.635999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":417792,"data_stored":155463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":9.2400000000000002,"5min":9.4960000000000004,"15min":9.5380000000000003},"min":{"1min":6.1849999999999996,"5min":5.0449999999999999,"15min":5.0449999999999999},"max":{"1min":15.425000000000001,"5min":20.558,"15min":20.558},"last":11.228},{"interface":"front","average":{"1min":12.69,"5min":12.125,"15min":12.031000000000001},"min":{"1min":8.2319999999999993,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":16.902000000000001,"5min":20.893999999999998,"15min":20.893999999999998},"last":15.747}]},{"osd":1,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":12.448,"5min":15.959,"15min":16.544},"min":{"1min":7.7889999999999997,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":20.911000000000001,"5min":51.890000000000001,"15min":51.890000000000001},"last":14.869999999999999},{"interface":"front","average":{"1min":8.5869999999999997,"5min":8.4309999999999992,"15min":8.4049999999999994},"min":{"1min":4.7519999999999998,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":21.609999999999999,"5min":35.075000000000003,"15min":35.075000000000003},"last":9.3970000000000002}]},{"osd":2,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":13.022,"5min":12.69,"15min":12.635},"min":{"1min":6.6440000000000001,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":30.649999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":12.885999999999999},{"interface":"front","average":{"1min":12.874000000000001,"5min":11.499000000000001,"15min":11.27},"min":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":23.489999999999998,"5min":23.489999999999998,"15min":23.489999999999998},"last":16.853999999999999}]},{"osd":3,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":27.213999999999999,"15min":29.18},"min":{"1min":7.9829999999999997,"5min":7.3140000000000001,"15min":7.3140000000000001},"max":{"1min":32.116,"5min":228.17099999999999,"15min":228.17099999999999},"last":7.0739999999999998},{"interface":"front","average":{"1min":15.337999999999999,"5min":29.597000000000001,"15min":31.972999999999999},"min":{"1min":10.156000000000001,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":26.178000000000001,"5min":270.13,"15min":270.13},"last":11.853999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":14.018000000000001,"5min":16.728000000000002,"15min":17.18},"min":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":25.247,"5min":86.980999999999995,"15min":86.980999999999995},"last":23.157},{"interface":"front","average":{"1min":17.876000000000001,"5min":20.236000000000001,"15min":20.629000000000001},"min":{"1min":9.2059999999999995,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":31.477,"5min":99.021000000000001,"15min":99.021000000000001},"last":7.8719999999999999}]},{"osd":5,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":20.492000000000001,"5min":21.867999999999999,"15min":22.097999999999999},"min":{"1min":10.611000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"max":{"1min":48.439,"5min":97.191000000000003,"15min":97.191000000000003},"last":10.254},{"interface":"front","average":{"1min":18.427,"5min":20.876999999999999,"15min":21.285},"min":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":49.709000000000003,"5min":64.048000000000002,"15min":64.048000000000002},"last":25.257000000000001}]},{"osd":6,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.638999999999999,"5min":17.018999999999998,"15min":17.416},"min":{"1min":9.3729999999999993,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":19.253,"5min":74.064999999999998,"15min":74.064999999999998},"last":22.521000000000001},{"interface":"front","average":{"1min":19.581,"5min":22.021000000000001,"15min":22.427},"min":{"1min":11.448,"5min":11.448,"15min":11.448},"max":{"1min":34.610999999999997,"5min":92.477999999999994,"15min":92.477999999999994},"last":14.013999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T07:59:35.015 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [4, 3] 2023-12-06T07:59:35.016 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.0', '4', '3'] 2023-12-06T07:59:35.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.0 4 3 2023-12-06T07:59:35.054 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:35.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:35.060 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.060 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:35.155 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:35.153+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:35.236 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.237 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:35.255 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:35.254+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:35.356 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:35.354+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:35.373 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.373 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:35.457 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:35.456+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:35.513 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.513 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:35.566 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:35.565+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:35.649 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.649 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:35.660 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:35.658+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:35.760 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:35.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:35.829 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:35.829 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:35.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T07:59:35.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:35.879 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:35.879+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:35.963 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:35.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:36.005 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.006 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:36.064 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:36.063+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:36.143 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.143 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:36.164 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:36.164+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:36.265 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:36.265+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:36.281 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.281 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:36.370 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:36.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:36.417 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.418 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:36.468 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:36.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:36.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T07:59:36.554 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.555 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:36.569 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:36.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:36.669 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:36.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:36.692 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.692 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:36.771 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:36.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:36.830 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.831 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:36.883 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:36.882+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:36.970 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:36.970 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:36.972 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:36.970+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:36.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:37.072 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:37.072+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:37.151 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.151 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:37.204 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:37.203+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:37.274 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:37.273+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:37.288 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.288 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:37.382 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:37.381+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:37.464 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.464 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:37.514 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:37.513+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:37.577 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:37.576+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:37.642 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.642 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:37.678 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:37.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:37.780 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:37.779+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:37.782 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.782 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:37.881 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:37.880+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:37.919 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:37.920 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:37.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:37.982 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:37.981+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:38.069 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.069 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:38.082 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:38.082+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:38.183 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:38.183+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:38.206 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.207 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:38.284 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:38.283+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:38.348 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.348 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:38.385 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:38.384+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:38.485 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:38.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:38.525 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.526 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:38.619 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:38.618+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:38.687 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:38.687+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:38.702 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.702 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:38.788 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:38.788+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:38.882 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:38.882 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:38.889 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:38.889+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:38.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:38.990 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:38.989+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:39.060 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.060 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:59:39.094 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T07:59:39.094 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T07:59:39.095 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T07:59:39.095 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:59:39.109 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T07:59:39.098+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 97381) UID: 0 2023-12-06T07:59:39.109 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T07:59:39.099+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T07:59:39.113 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:39.113+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:39.194 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:39.194+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:39.201 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.201 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:39.295 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:39.292+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:39.382 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.382 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:39.412 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:39.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:39.495 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:39.495+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:39.552 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.552 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:39.595 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:39.595+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:39.644 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:06.434 97394== 2023-12-06T07:59:39.644 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:06.434 97394== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:59:39.656 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:59:39.703 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:39.703+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:39.707 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.708 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:39.797 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:39.797+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:39.844 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.845 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:39.898 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:39.898+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:39.982 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:39.983 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:39.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:39.998 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:39.998+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:40.099 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:40.099+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:40.118 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.118 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:40.201 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:40.201+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:40.255 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.255 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:40.302 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:40.302+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:40.397 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.397 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.403 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:40.402+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:40.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:40.504 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:40.504+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:40.537 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.538 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:40.605 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:40.605+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:40.676 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.676 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:40.706 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:40.706+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:40.807 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:40.807+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:40.855 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:40.855 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:40.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:40.908 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:40.908+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:41.009 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:41.009+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:41.036 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.036 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:41.110 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:41.109+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:41.211 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:41.211+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:41.216 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.216 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:41.312 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:41.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:41.353 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.354 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:41.448 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:41.448+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:41.514 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:41.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:41.535 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.535 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:41.617 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:41.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:41.713 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.713 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.717 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:41.717+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:41.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:41.818 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:41.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:41.863 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:41.863 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:41.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:41.958 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:41.957+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:42.021 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:42.021+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:42.044 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.044 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:42.122 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:42.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:42.223 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:42.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:42.224 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.224 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:42.324 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:42.324+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:42.405 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.405 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:42.425 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:42.425+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:42.526 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:42.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:42.543 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.543 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:42.628 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:42.628+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:42.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T07:59:42.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T07:59:42.680 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.680 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:42.735 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:42.734+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:42.820 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.820 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:42.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:42.832 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:42.830+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:42.931 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:42.931+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:42.998 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:42.998 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:43.032 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:43.032+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:43.135 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:43.134+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:43.135 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.135 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:43.236 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:43.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:43.317 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.317 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:43.371 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:43.370+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:43.440 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:43.440+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:43.454 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.455 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:43.542 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:43.540+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:43.631 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:43.641 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:43.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:43.742 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:43.742+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:43.773 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.773 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:43.845 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:43.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:43.909 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:43.909 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:43.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:43.959 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:43.959+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:44.046 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:44.046+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:44.087 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.087 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:44.148 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:44.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:44.249 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:44.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:44.265 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.266 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:44.350 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:44.350+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:44.405 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.405 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:44.450 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:44.450+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:44.552 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:44.552+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:44.582 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.583 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:44.653 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:44.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:44.753 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:44.753+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:44.760 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.760 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:44.855 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:44.855+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:44.939 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:44.940 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:44.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:44.992 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:44.992+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:45.062 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:45.062+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:45.076 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.076 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:45.158 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:45.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:59:45.197 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T07:59:45.197 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T07:59:45.197 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T07:59:45.198 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T07:59:45.214 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T07:59:45.202+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 94554) UID: 0 2023-12-06T07:59:45.214 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T07:59:45.203+0000 9f54640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2023-12-06T07:59:45.258 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.258 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.259 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:45.258+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:45.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:45.360 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:45.360+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:45.439 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.439 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:45.461 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:45.461+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:45.563 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:45.562+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:45.586 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.586 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:45.664 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:45.664+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:45.720 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:44.567 94560== 2023-12-06T07:59:45.720 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:44.567 94560== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T07:59:45.731 INFO:tasks.ceph.mon.a.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T07:59:45.736 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.736 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:45.790 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:45.790+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:45.867 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:45.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:45.877 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:45.877 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:45.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:45.972 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:45.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:46.055 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.056 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:46.107 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:46.106+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:46.170 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:46.170+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:46.238 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.238 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:46.272 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:46.272+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:46.372 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:46.372+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:46.416 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.416 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:46.510 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:46.509+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:46.573 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:46.573+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:46.595 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.595 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:46.690 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:46.689+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:46.774 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.774 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.774 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:46.774+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:46.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:46.875 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:46.875+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:46.956 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:46.956 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:46.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:46.976 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:46.976+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:47.076 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:47.076+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:47.097 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.097 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:47.178 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:47.178+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:47.279 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:47.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:47.284 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.284 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:47.379 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:47.379+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:47.461 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.461 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:47.480 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:47.480+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:47.581 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:47.581+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:47.598 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.598 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:47.682 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:47.682+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:47.743 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.743 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:47.784 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:47.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:47.885 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:47.885+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:47.919 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:47.920 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:47.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:47.985 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:47.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:48.055 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.055 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:48.086 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:48.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:48.187 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:48.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:48.232 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.232 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:48.288 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:48.287+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:48.396 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:48.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:48.410 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.410 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:48.504 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:48.504+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:48.588 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.588 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.592 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:48.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:48.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:48.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T07:59:48.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T07:59:48.692 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:48.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:48.766 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.766 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:48.817 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:48.817+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:48.895 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:48.895+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:48.943 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:48.944 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:48.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:48.996 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:48.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:49.097 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:49.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:49.122 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.123 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:49.199 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:49.199+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:49.258 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.258 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:49.309 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:49.309+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:49.422 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:49.422+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:49.436 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.437 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:49.530 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:49.530+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:49.603 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:49.603+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:49.614 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.614 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:49.704 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:49.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:49.766 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.766 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:49.821 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:49.820+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:49.906 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:49.906+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:49.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:50.007 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:50.007+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:50.084 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.084 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:50.138 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:50.137+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:50.209 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:50.209+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:50.223 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.223 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:50.311 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:50.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:50.398 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.398 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:50.411 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:50.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:50.512 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:50.511+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:50.535 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.536 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:50.613 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:50.613+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:50.712 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.712 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.713 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:50.713+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:50.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:50.814 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:50.814+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:50.891 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:50.892 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:50.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:50.915 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:50.915+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:51.016 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:51.016+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:51.070 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.070 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:51.122 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:51.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:51.218 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:51.218+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:51.252 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.252 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T07:59:51.299 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T07:59:51.300 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T07:59:51.300 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2023-12-06T07:59:51.300 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T07:59:51.300 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T07:59:51.306 INFO:tasks.ceph.mon.b:Started 2023-12-06T07:59:51.306 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T07:59:51.307 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T07:59:51.309 INFO:tasks.ceph.mon.c:Started 2023-12-06T07:59:51.309 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T07:59:51.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T07:59:51.354 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:51.353+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:51.391 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.391 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:51.420 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:51.420+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:51.523 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:51.523+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:51.572 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.572 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:51.626 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:51.626+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:51.714 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.788 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:51.790 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:51.790+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:51.890 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:51.890+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:51.960 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:51.960 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:51.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:51.991 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:51.991+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:52.092 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:52.092+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:52.103 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.103 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:52.193 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:52.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:52.244 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.244 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:52.294 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:52.294+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:52.386 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.387 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.395 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:52.395+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:52.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:52.496 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:52.497+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:52.528 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.528 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:52.598 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:52.598+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:52.686 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.686 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:52.698 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:52.698+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:52.798 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:52.798+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:52.845 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:52.846 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:52.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:52.899 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:52.899+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:53.000 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:53.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:53.041 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.041 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:53.100 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:53.100+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:53.180 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.181 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:53.201 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:53.202+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:53.303 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:53.302+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:53.362 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.362 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:53.404 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:53.404+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:53.526 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:53.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:53.540 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.541 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:53.606 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:53.607+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:53.682 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.682 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:53.707 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:53.707+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:53.808 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:53.808+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:53.863 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:53.863 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:53.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:53.909 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:53.908+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:54.009 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:54.010+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:54.045 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.045 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:54.140 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:54.140+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:54.211 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:54.212+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:54.238 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.238 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:54.313 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:54.313+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:54.414 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:54.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:54.419 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.419 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:54.515 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:54.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:54.560 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.561 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:54.655 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:54.655+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:54.717 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:54.718+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:54.740 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.740 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:54.834 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:54.834+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:54.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T07:59:54.919 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:54.919+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:54.920 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:54.920 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:54.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:55.019 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:55.020+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:55.103 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.104 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:55.121 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:55.120+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:55.222 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:55.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:55.340 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:55.340+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:55.424 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:55.424+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:55.469 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.469 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:55.525 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:55.526+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:55.609 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.609 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:55.664 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:55.664+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:55.726 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:55.726+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:55.750 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.750 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:55.827 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:55.828+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:55.928 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:55.928+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:55.933 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:55.933 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:55.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:56.029 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:56.029+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:56.112 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.112 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:56.167 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:56.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:56.233 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:56.231+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:56.252 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.253 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T07:59:56.332 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:56.332+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:56.432 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.432 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.432 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:56.433+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:56.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T07:59:56.533 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:56.534+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:56.571 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.571 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T07:59:56.634 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:56.634+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:56.710 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.711 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T07:59:56.735 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:56.735+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:56.836 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:56.836+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:56.851 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.851 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:56.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T07:59:56.941 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T07:59:56.939+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T07:59:56.995 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:56.995 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T07:59:57.049 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:57.049+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:57.135 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.135 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.138 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:57.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:57.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T07:59:57.240 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:57.240+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:57.314 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.314 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T07:59:57.365 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:57.365+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:57.441 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:57.441+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:57.494 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.494 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T07:59:57.549 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:57.549+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T07:59:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.635 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.642 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:57.642+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:57.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T07:59:57.743 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:57.743+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:57.813 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.814 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T07:59:57.866 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:57.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:57.945 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:57.946+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:57.951 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:57.951 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:57.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T07:59:58.046 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:58.046+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:58.129 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.129 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T07:59:58.147 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:58.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:58.248 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:58.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:58.273 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.274 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T07:59:58.348 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:58.348+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:58.411 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.411 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T07:59:58.465 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:58.465+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:58.550 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:58.549+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:58.566 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.566 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T07:59:58.650 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:58.650+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:58.746 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.747 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.751 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:58.751+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:58.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T07:59:58.852 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T07:59:58.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T07:59:58.885 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:58.885 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:58.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T07:59:58.953 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:58.954+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:59.035 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.036 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T07:59:59.088 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:59.088+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:59.155 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T07:59:59.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T07:59:59.216 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.216 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T07:59:59.256 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:59.256+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:59.301 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:07.934 97689== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:59.356 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.356 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.357 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T07:59:59.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T07:59:59.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T07:59:59.382 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.002 104678== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:59.457 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T07:59:59.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T07:59:59.497 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.497 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T07:59:59.531 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:08.164 97689== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:59.559 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:59.559+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:59.624 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.244 104678== Warning: unimplemented fcntl command: 1036 2023-12-06T07:59:59.660 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:59.659+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:59.676 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.676 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T07:59:59.771 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T07:59:59.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T07:59:59.857 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T07:59:59.857 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T07:59:59.861 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T07:59:59.862+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T07:59:59.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T07:59:59.962 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T07:59:59.963+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:00.038 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.038 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:00.063 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:00.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:00.164 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:00.164+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:00.179 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.179 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:00.265 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:00.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:00.359 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.360 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.366 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:00.366+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:00.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:00.468 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:00.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:00.540 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.541 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:00.568 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:00.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:00.669 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:00.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:00.679 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.680 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:00.770 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:00.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:00.872 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:00.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:00.878 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:00.878 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:00.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:00.974 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:00.973+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:01.060 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.060 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:01.074 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:01.075+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:01.175 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:01.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:01.240 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.240 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:01.276 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:01.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:01.377 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:01.377+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:01.379 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.379 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:01.479 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:01.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:01.522 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.522 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:01.579 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:01.579+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:01.680 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:01.680+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:01.702 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.702 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:01.782 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:01.782+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:01.840 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.840 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:01.894 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:01.894+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:01.984 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:01.984 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:01.984 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:01.984+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:01.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:02.085 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:02.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:02.167 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:02.169 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:02.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:02.217 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:02.218+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:02.287 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:02.287+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:02.347 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:02.347 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:02.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:02.388 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:02.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:02.489 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:02.489+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:02.489 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:02.489 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:02.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:02.590 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:02.590+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:02.646 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:02.646 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:02.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:02.698 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:02.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:02.792 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:02.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:02.848 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:02.848 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:02.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:02.899 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:02.900+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:02.993 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:02.993+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:03.055 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.055 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:03.109 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:03.109+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:03.196 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:03.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:03.215 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.215 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:03.310 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:03.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:03.398 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:03.397+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:03.446 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.446 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:03.500 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:03.500+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:03.582 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.582 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:03.602 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:03.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:03.703 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:03.703+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:03.761 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.761 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:03.813 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:03.813+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:03.906 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:03.906+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:03.941 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:03.941 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:03.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:04.006 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:04.007+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:04.076 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.076 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:04.106 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:04.107+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:04.208 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:04.208+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:04.213 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.213 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:04.312 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:04.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:04.350 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.350 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:04.413 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:04.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:04.519 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:04.519+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:04.527 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.528 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:04.621 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:04.621+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:04.705 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.705 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:04.758 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:04.756+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:04.826 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:04.826+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:04.847 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:04.847 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:04.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:04.924 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:04.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:05.024 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:05.024+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:05.027 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.028 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:05.125 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:05.125+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:05.206 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.207 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:05.226 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:05.226+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:05.326 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:05.327+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:05.343 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.343 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:05.428 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:05.428+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:05.496 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.496 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:05.550 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:05.550+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:05.629 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:05.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:05.633 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.634 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:05.730 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:05.731+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:05.812 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.812 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:05.831 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:05.831+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:05.932 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:05.932+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:05.989 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:05.989 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:05.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:06.033 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:06.033+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:06.126 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.126 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.134 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:06.135+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:06.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:06.236 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:06.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:06.308 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.308 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:06.359 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:06.359+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:06.439 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:06.438+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:06.486 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.486 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:06.540 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:06.540+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:06.621 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.621 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:06.641 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:06.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:06.742 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:06.742+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:06.800 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.800 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:06.860 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:06.859+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:06.941 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:06.941 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:06.942 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:06.943+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:06.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:07.049 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:07.043+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:07.118 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.119 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:07.145 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:07.144+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:07.246 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:07.246+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:07.266 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.266 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:07.347 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:07.347+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:07.416 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.417 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:07.468 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:07.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:07.549 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:07.549+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:07.595 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.595 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:07.650 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:07.649+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:07.735 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.735 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:07.787 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:07.787+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:07.853 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:07.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:07.872 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:07.872 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:07.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:07.956 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:07.955+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:08.056 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.056 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.057 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:08.056+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:08.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:08.158 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:08.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:08.195 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.195 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:08.259 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:08.259+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:08.360 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:08.360+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:08.393 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.393 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:08.487 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:08.487+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:08.562 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:08.562+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:08.591 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.591 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:08.686 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:08.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:08.770 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:08.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:08.773 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.774 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:08.871 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:08.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:08.957 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:08.957 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:08.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:09.005 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:09.005+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:09.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:09.021 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:09.073 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:09.073+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:09.111 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:09.112 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:09.151 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:09.152 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:09.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:09.174 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:09.173+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:09.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:09.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:09.275 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:09.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:09.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:09.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:09.360 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:09.360 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:09.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:09.360 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:09.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 149.943274057, 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.361 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:09.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:09.365 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:09.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:09.411 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:09.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:09.427 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:09.427 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:09.476 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:09.476+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:09.503 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:09.503 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:09.503 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:09.503 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:09.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:09.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:09.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:09.577 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:09.577+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:09.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:09.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:09.678 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:09.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:09.699 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:09.699 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:09.699 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:09.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:09.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:09.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:09.780 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:09.779+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:09.837 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:09.837 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:09.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:09.837 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:09.837 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:09.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:09.881 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:09.880+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:09.981 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:09.981+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:10.032 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:10.033 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:10.033 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:10.033 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:10.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:10.083 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:10.083+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:10.086 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:10.100 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T08:00:10.100 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T08:00:10.100 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T08:00:10.137 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.0 pg_upmap mapping to [4,3] 2023-12-06T08:00:10.139 INFO:tasks.ceph.mon.a:Started 2023-12-06T08:00:10.139 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T08:00:10.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:10.139 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:00:10.268 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:10.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:10.287 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:10.287+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:10.319 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:10.319 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:10.319 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:10.319 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:10.319 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:10.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:10.390 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:10.390+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:10.491 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:10.491+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:10.495 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:10.495 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:10.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 169.51572341900001, 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:10.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:10.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.85194241799999, 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.848661789, 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:10.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:10.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:10.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.508 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.513 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.54388517300001, 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.54304207600001, 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.52050657000001, 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:10.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:10.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:10.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.51851415100001, 2023-12-06T08:00:10.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:10.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.530 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.517981886, 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:10.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:10.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:10.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.50105697699999, 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.543 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 168.500220343, 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:10.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:10.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.47577307500001, 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.469870711, 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.46516471499999, 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.46286284000001, 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.564 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.462364517, 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.461910606, 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:10.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.572 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 167.46068820599999, 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:10.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:10.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:10.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:10.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 161.815522225, 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.577 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:10.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 161.80808381, 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 161.79801043800001, 2023-12-06T08:00:10.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:10.589 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:10.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:10.593 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:10.592+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:10.693 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:10.693+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:10.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:10.829 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:10.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:10.868 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:10.869 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:10.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:10.926 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:10.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:10.995 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:10.995+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:11.018 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:11.019 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:11.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:11.019 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:11.019 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:11.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:11.112 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:11.111+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:11.197 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:11.197 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:11.197 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:11.197 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:11.198 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:11.197+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:11.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:11.300 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:11.299+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:11.377 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:11.378 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:11.378 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:11.378 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:11.378 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:11.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:11.428 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:11.428+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:11.502 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:11.502+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:11.558 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 150.98215825400001, 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:11.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:11.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:11.562 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:11.563 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:11.563 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:11.563 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:11.563 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:11.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:11.596 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:20.217 104678== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.609 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:11.608+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:11.651 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:20.271 104678== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.674 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.307 97689== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.704 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:11.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:11.719 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:11.719 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:11.719 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:11.719 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:11.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:00:11.730 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.359 97689== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:11.805 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:11.805+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:11.877 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:20.498 104678== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.906 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:11.906+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:11.908 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:20.542 97689== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:11.910 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:11.911 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:11.911 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:11.911 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:11.911 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:11.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:12.007 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:12.006+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:12.108 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:12.107+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:12.112 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:12.112 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:12.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:12.112 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:12.112 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:12.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:12.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:00:12.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:12.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:12.209 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:12.208+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:12.310 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:12.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:12.338 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:12.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:12.339 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:12.339 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:12.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:12.411 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:12.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:12.427 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:12.427 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:12.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:12.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:12.514 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:12.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:12.522 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:12.522 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:12.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:12.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:12.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:12.615 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:12.615+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:12.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:12.692 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:12.716 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:12.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:12.725 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:12.725 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:12.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:12.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:12.817 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:12.817+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:12.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:12.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:12.918 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:12.917+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:12.932 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:12.932 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:12.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:13.019 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:13.018+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:13.111 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.111 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.120 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:13.119+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:13.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:13.222 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:13.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:13.255 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.255 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:13.349 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:13.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:13.425 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:13.425+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:13.434 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.434 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:13.528 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:13.528+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:13.614 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.614 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:13.665 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:13.665+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:13.728 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:13.727+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:13.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:13.794 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.794 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:13.850 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:13.849+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:13.893 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:00:13.930 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:13.929+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:13.945 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:13.945 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:13.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:14.038 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:14.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:14.122 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.122 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:14.132 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:14.131+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:14.232 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:14.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:14.311 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.311 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:14.365 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:14.364+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:14.435 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:14.435+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:14.450 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:14.452 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.453 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:14.538 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:14.538+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:14.635 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.635 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.638 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:14.638+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:14.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:14.739 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:14.739+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:14.812 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.812 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:14.862 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:14.861+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:14.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:00:14.941 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:14.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:14.989 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:14.989 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:14.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:15.042 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:15.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:15.143 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:15.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:15.143 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:15.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:15.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:00:15.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:00:15.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:00:15.165 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:15.165 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:15.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:15.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:15.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:15.244 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:15.244+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:15.346 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:15.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:15.349 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:15.349 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:15.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:15.381 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:15.382 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:15.446 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:15.446+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:15.467 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:15.468 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:15.533 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:15.533 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:15.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:15.547 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:15.547+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:15.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:15.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:15.648 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:15.648+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:15.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:15.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:15.714 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:15.715 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:15.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:15.732 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:00:15.733 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:00:15.749 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2023-12-06T08:00:13.830524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220384,"kb_used_data":4216,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754336,"statfs":{"total":773094113280,"available":772868440064,"internally_reserved":0,"allocated":4317184,"data_stored":2101416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000998"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:40.832806+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T07:59:40.832806+0000","last_peered":"2023-12-06T07:59:40.832806+0000","last_clean":"2023-12-06T07:59:40.832806+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T07:59:40.832806+0000","last_undegraded":"2023-12-06T07:59:40.832806+0000","last_fullsized":"2023-12-06T07:59:40.832806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:31.660428+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T07:59:31.660428+0000","last_peered":"2023-12-06T07:59:31.660428+0000","last_clean":"2023-12-06T07:59:31.660428+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T07:59:31.660428+0000","last_undegraded":"2023-12-06T07:59:31.660428+0000","last_fullsized":"2023-12-06T07:59:31.660428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-06T07:59:28.085605+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T07:59:28.085605+0000","last_peered":"2023-12-06T07:59:28.085605+0000","last_clean":"2023-12-06T07:59:28.085605+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T07:59:28.085605+0000","last_undegraded":"2023-12-06T07:59:28.085605+0000","last_fullsized":"2023-12-06T07:59:28.085605+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:36.840025+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T07:59:36.840025+0000","last_peered":"2023-12-06T07:59:36.840025+0000","last_clean":"2023-12-06T07:59:36.840025+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T07:59:36.840025+0000","last_undegraded":"2023-12-06T07:59:36.840025+0000","last_fullsized":"2023-12-06T07:59:36.840025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:35.828242+0000","last_change":"2023-12-06T07:57:30.917200+0000","last_active":"2023-12-06T07:59:35.828242+0000","last_peered":"2023-12-06T07:59:35.828242+0000","last_clean":"2023-12-06T07:59:35.828242+0000","last_became_active":"2023-12-06T07:57:30.912077+0000","last_became_peered":"2023-12-06T07:57:30.912077+0000","last_unstale":"2023-12-06T07:59:35.828242+0000","last_undegraded":"2023-12-06T07:59:35.828242+0000","last_fullsized":"2023-12-06T07:59:35.828242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:48:52.094966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:35.647427+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T07:59:35.647427+0000","last_peered":"2023-12-06T07:59:35.647427+0000","last_clean":"2023-12-06T07:59:35.647427+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T07:59:35.647427+0000","last_undegraded":"2023-12-06T07:59:35.647427+0000","last_fullsized":"2023-12-06T07:59:35.647427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-12-06T08:00:11.374225+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:00:11.374225+0000","last_peered":"2023-12-06T08:00:11.374225+0000","last_clean":"2023-12-06T08:00:11.374225+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:00:11.374225+0000","last_undegraded":"2023-12-06T08:00:11.374225+0000","last_fullsized":"2023-12-06T08:00:11.374225+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-06T07:59:35.828609+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T07:59:35.828609+0000","last_peered":"2023-12-06T07:59:35.828609+0000","last_clean":"2023-12-06T07:59:35.828609+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T07:59:35.828609+0000","last_undegraded":"2023-12-06T07:59:35.828609+0000","last_fullsized":"2023-12-06T07:59:35.828609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":141,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-06T07:59:28.081717+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T07:59:28.081717+0000","last_peered":"2023-12-06T07:59:28.081717+0000","last_clean":"2023-12-06T07:59:28.081717+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T07:59:28.081717+0000","last_undegraded":"2023-12-06T07:59:28.081717+0000","last_fullsized":"2023-12-06T07:59:28.081717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476773,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":164339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":9.5679999999999996,"5min":9.5860000000000003,"15min":9.5890000000000004},"min":{"1min":5.202,"5min":5.202,"15min":5.202},"max":{"1min":27.638999999999999,"5min":27.638999999999999,"15min":27.638999999999999},"last":8.5139999999999993},{"interface":"front","average":{"1min":9.7639999999999993,"5min":8.8399999999999999,"15min":8.6859999999999999},"min":{"1min":5.0309999999999997,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":24.085000000000001,"5min":24.085000000000001,"15min":24.085000000000001},"last":10.052}]},{"osd":1,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":10.199,"5min":12.532999999999999,"15min":12.922000000000001},"min":{"1min":6.3970000000000002,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":21.231999999999999,"5min":29.213000000000001,"15min":29.213000000000001},"last":11.167},{"interface":"front","average":{"1min":13.334,"5min":14.449999999999999,"15min":14.635999999999999},"min":{"1min":9.0530000000000008,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":23.311,"5min":39.337000000000003,"15min":39.337000000000003},"last":17.731999999999999}]},{"osd":2,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":11.083,"5min":12.324999999999999,"15min":12.532},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":17.152000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":14.218},{"interface":"front","average":{"1min":14.012,"5min":14.178000000000001,"15min":14.206},"min":{"1min":7.4470000000000001,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":26.388000000000002,"5min":26.388000000000002,"15min":26.388000000000002},"last":15.529999999999999}]},{"osd":4,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":15.282,"5min":13.589,"15min":13.307},"min":{"1min":10.794,"5min":8.6259999999999994,"15min":8.6259999999999994},"max":{"1min":20.471,"5min":23.986000000000001,"15min":23.986000000000001},"last":20.997},{"interface":"front","average":{"1min":16.187000000000001,"5min":16.135999999999999,"15min":16.128},"min":{"1min":8.0500000000000007,"5min":7.806,"15min":7.806},"max":{"1min":54.225999999999999,"5min":54.225999999999999,"15min":54.225999999999999},"last":27.959}]},{"osd":5,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":16.652999999999999,"5min":17.109000000000002,"15min":17.184999999999999},"min":{"1min":9.6880000000000006,"5min":8.8580000000000005,"15min":8.8580000000000005},"max":{"1min":29.523,"5min":29.523,"15min":29.523},"last":30.466000000000001},{"interface":"front","average":{"1min":15.603,"5min":17.038,"15min":17.277000000000001},"min":{"1min":8.3870000000000005,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":21.338999999999999,"5min":38.670000000000002,"15min":38.670000000000002},"last":32.436}]},{"osd":6,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":12.997999999999999,"5min":14.481,"15min":14.728},"min":{"1min":6.3680000000000003,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":27.890000000000001,"5min":27.890000000000001,"15min":27.890000000000001},"last":16.481999999999999},{"interface":"front","average":{"1min":19.378,"5min":18.928000000000001,"15min":18.853000000000002},"min":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":44.255000000000003,"5min":44.255000000000003,"15min":44.255000000000003},"last":23.527000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:59:23 2023","interfaces":[{"interface":"back","average":{"1min":17.896000000000001,"5min":19.148,"15min":19.356999999999999},"min":{"1min":12.214,"5min":12.214,"15min":12.214},"max":{"1min":23.091000000000001,"5min":40.832000000000001,"15min":40.832000000000001},"last":12.896000000000001},{"interface":"front","average":{"1min":17.309000000000001,"5min":18.459,"15min":18.651},"min":{"1min":10.505000000000001,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":29.32,"5min":39.331000000000003,"15min":39.331000000000003},"last":18.956}]}]},{"osd":2,"up_from":16,"seq":68719476774,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":159872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":8.9700000000000006,"5min":10.191000000000001,"15min":10.395},"min":{"1min":4.5359999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":17.594999999999999,"5min":22.759,"15min":22.759},"last":12.487},{"interface":"front","average":{"1min":10.936999999999999,"5min":10.676,"15min":10.632},"min":{"1min":5.9180000000000001,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":30.768999999999998,"5min":34.981000000000002,"15min":34.981000000000002},"last":6.3579999999999997}]},{"osd":1,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":13.324999999999999,"5min":20.66,"15min":21.882000000000001},"min":{"1min":7.6429999999999998,"5min":6.7800000000000002,"15min":6.7800000000000002},"max":{"1min":27.763000000000002,"5min":165.68600000000001,"15min":165.68600000000001},"last":15.388},{"interface":"front","average":{"1min":11.452999999999999,"5min":20.914999999999999,"15min":22.492000000000001},"min":{"1min":6.3760000000000003,"5min":6.3760000000000003,"15min":6.3760000000000003},"max":{"1min":27.297999999999998,"5min":153.68199999999999,"15min":153.68199999999999},"last":8.6189999999999998}]},{"osd":3,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":8.9290000000000003,"5min":10.803000000000001,"15min":11.115},"min":{"1min":5.4299999999999997,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":14.581,"5min":35.935000000000002,"15min":35.935000000000002},"last":10.861000000000001},{"interface":"front","average":{"1min":16.529,"5min":15.874000000000001,"15min":15.765000000000001},"min":{"1min":9.4629999999999992,"5min":8.6560000000000006,"15min":8.6560000000000006},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":13.182}]},{"osd":4,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":11.83,"5min":15.986000000000001,"15min":16.678999999999998},"min":{"1min":7.0830000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":17.033000000000001,"5min":50.137999999999998,"15min":50.137999999999998},"last":11.868},{"interface":"front","average":{"1min":17.085000000000001,"5min":16.704999999999998,"15min":16.641999999999999},"min":{"1min":9.9350000000000005,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":31.765000000000001,"5min":46.325000000000003,"15min":46.325000000000003},"last":17.044}]},{"osd":5,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":19.056000000000001,"5min":20.315999999999999,"15min":20.526},"min":{"1min":10.106,"5min":10.106,"15min":10.106},"max":{"1min":32.866,"5min":50.320999999999998,"15min":50.320999999999998},"last":17.934999999999999},{"interface":"front","average":{"1min":14.555,"5min":17.138999999999999,"15min":17.568999999999999},"min":{"1min":7.9370000000000003,"5min":7.9370000000000003,"15min":7.9370000000000003},"max":{"1min":25.809999999999999,"5min":44.323,"15min":44.323},"last":7.9820000000000002}]},{"osd":6,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":15.962999999999999,"5min":19.661999999999999,"15min":20.277999999999999},"min":{"1min":9.2330000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":48.231000000000002,"5min":104.834,"15min":104.834},"last":13.885},{"interface":"front","average":{"1min":22.042999999999999,"5min":21.042999999999999,"15min":20.876000000000001},"min":{"1min":12.053000000000001,"5min":12.053000000000001,"15min":12.053000000000001},"max":{"1min":49.146999999999998,"5min":49.146999999999998,"15min":49.146999999999998},"last":18.972999999999999}]},{"osd":7,"last update":"Wed Dec 6 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":17.242999999999999,"5min":19.402999999999999,"15min":19.763000000000002},"min":{"1min":10.637,"5min":7.8440000000000003,"15min":7.8440000000000003},"max":{"1min":27.006,"5min":42.915999999999997,"15min":42.915999999999997},"last":10.212},{"interface":"front","average":{"1min":16.879999999999999,"5min":18.922000000000001,"15min":19.262},"min":{"1min":9.8239999999999998,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":26.298999999999999,"5min":42.506,"15min":42.506},"last":14.472}]}]},{"osd":1,"up_from":16,"seq":68719476774,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":557712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":9.718,"5min":12.887,"15min":13.414999999999999},"min":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":5.0890000000000004},"max":{"1min":19.373999999999999,"5min":80.640000000000001,"15min":80.640000000000001},"last":8.2119999999999997},{"interface":"front","average":{"1min":8.9740000000000002,"5min":11.124000000000001,"15min":11.481999999999999},"min":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"max":{"1min":18.952000000000002,"5min":25.390000000000001,"15min":25.390000000000001},"last":12.407}]},{"osd":2,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.468,"5min":19.667000000000002,"15min":20.699999999999999},"min":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.533000000000001,"5min":100.703,"15min":100.703},"last":9.3829999999999991},{"interface":"front","average":{"1min":13.167999999999999,"5min":18.32,"15min":19.178999999999998},"min":{"1min":6.532,"5min":6.532,"15min":6.532},"max":{"1min":23.036999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":7.0789999999999997}]},{"osd":3,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":14.539,"5min":17.850999999999999,"15min":18.402999999999999},"min":{"1min":8.0250000000000004,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":29.652999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":13.013999999999999},{"interface":"front","average":{"1min":9.5229999999999997,"5min":13.081,"15min":13.673999999999999},"min":{"1min":6.3390000000000004,"5min":6.3390000000000004,"15min":6.3390000000000004},"max":{"1min":15.93,"5min":46.613,"15min":46.613},"last":6.7880000000000003}]},{"osd":4,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":15.394,"5min":18.117999999999999,"15min":18.571999999999999},"min":{"1min":8.4450000000000003,"5min":8.4450000000000003,"15min":8.4450000000000003},"max":{"1min":23.760999999999999,"5min":81.838999999999999,"15min":81.838999999999999},"last":10.003},{"interface":"front","average":{"1min":16.411999999999999,"5min":20.725999999999999,"15min":21.445},"min":{"1min":10.313000000000001,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":27.658000000000001,"5min":66.515000000000001,"15min":66.515000000000001},"last":17.968}]},{"osd":5,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":12.662000000000001,"5min":15.837999999999999,"15min":16.367999999999999},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":28.388000000000002,"5min":56.235999999999997,"15min":56.235999999999997},"last":8.7929999999999993},{"interface":"front","average":{"1min":17.988,"5min":19.594000000000001,"15min":19.861999999999998},"min":{"1min":9.766,"5min":9.766,"15min":9.766},"max":{"1min":27.745000000000001,"5min":82.381,"15min":82.381},"last":11.118}]},{"osd":6,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":17.834,"15min":18.564},"min":{"1min":9.1959999999999997,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":29.748000000000001,"5min":58.619,"15min":58.619},"last":13.725},{"interface":"front","average":{"1min":18.754000000000001,"5min":20.972000000000001,"15min":21.341999999999999},"min":{"1min":9.6270000000000007,"5min":9.6229999999999993,"15min":9.6229999999999993},"max":{"1min":33.616999999999997,"5min":83.632000000000005,"15min":83.632000000000005},"last":10.532}]},{"osd":7,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":18.992000000000001,"5min":22.225999999999999,"15min":22.765000000000001},"min":{"1min":13.016,"5min":13.016,"15min":13.016},"max":{"1min":32.531999999999996,"5min":62.006999999999998,"15min":62.006999999999998},"last":18.574999999999999},{"interface":"front","average":{"1min":15.484999999999999,"5min":19.895,"15min":20.629999999999999},"min":{"1min":10.445,"5min":9.9740000000000002,"15min":9.9740000000000002},"max":{"1min":25.457000000000001,"5min":60.698999999999998,"15min":60.698999999999998},"last":11.815}]}]},{"osd":0,"up_from":16,"seq":68719476775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":159872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":9.1500000000000004,"5min":18.41,"15min":19.952999999999999},"min":{"1min":5.6470000000000002,"5min":5.6470000000000002,"15min":5.6470000000000002},"max":{"1min":19.452000000000002,"5min":252.88399999999999,"15min":252.88399999999999},"last":8.9930000000000003},{"interface":"front","average":{"1min":8.9000000000000004,"5min":19.289000000000001,"15min":21.021000000000001},"min":{"1min":5.9649999999999999,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":17.972000000000001,"5min":274.91500000000002,"15min":274.91500000000002},"last":11.683999999999999}]},{"osd":2,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":11.615,"5min":16.530999999999999,"15min":17.350999999999999},"min":{"1min":6.2649999999999997,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":20.821999999999999,"5min":82.866,"15min":82.866},"last":15.380000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":15.962999999999999,"15min":16.373999999999999},"min":{"1min":6.7930000000000001,"5min":6.7930000000000001,"15min":6.7930000000000001},"max":{"1min":20.969999999999999,"5min":85.991,"15min":85.991},"last":10.989000000000001}]},{"osd":3,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":10.926,"5min":14.554,"15min":15.157999999999999},"min":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":28.27,"5min":76.593000000000004,"15min":76.593000000000004},"last":7.7080000000000002},{"interface":"front","average":{"1min":13.904,"5min":19.632000000000001,"15min":20.585999999999999},"min":{"1min":6.9960000000000004,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":30.224,"5min":85.162000000000006,"15min":85.162000000000006},"last":16.061}]},{"osd":4,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":14.275,"5min":16.917000000000002,"15min":17.356999999999999},"min":{"1min":9.1389999999999993,"5min":7.516,"15min":7.516},"max":{"1min":22.143000000000001,"5min":65.349000000000004,"15min":65.349000000000004},"last":13.829000000000001},{"interface":"front","average":{"1min":10.933999999999999,"5min":13.465999999999999,"15min":13.888},"min":{"1min":6.5780000000000003,"5min":6.5780000000000003,"15min":6.5780000000000003},"max":{"1min":18.934000000000001,"5min":70.085999999999999,"15min":70.085999999999999},"last":12.422000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":18.460000000000001,"5min":26.390000000000001,"15min":27.712},"min":{"1min":9.8559999999999999,"5min":9.8559999999999999,"15min":9.8559999999999999},"max":{"1min":46.792000000000002,"5min":144.60499999999999,"15min":144.60499999999999},"last":17.515999999999998},{"interface":"front","average":{"1min":17.689,"5min":24.901,"15min":26.103000000000002},"min":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":45.198,"5min":142.25,"15min":142.25},"last":16.756}]},{"osd":6,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":13.087999999999999,"5min":17.364999999999998,"15min":18.077999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":17.132000000000001,"5min":75.471999999999994,"15min":75.471999999999994},"last":10.151999999999999},{"interface":"front","average":{"1min":16.850000000000001,"5min":22.954000000000001,"15min":23.971},"min":{"1min":9.4589999999999996,"5min":9.4589999999999996,"15min":9.4589999999999996},"max":{"1min":25.654,"5min":84.617000000000004,"15min":84.617000000000004},"last":19.02}]},{"osd":7,"last update":"Wed Dec 6 07:59:25 2023","interfaces":[{"interface":"back","average":{"1min":17.565000000000001,"5min":21.297000000000001,"15min":21.919},"min":{"1min":9.6890000000000001,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":28.094000000000001,"5min":84.031000000000006,"15min":84.031000000000006},"last":18.332000000000001},{"interface":"front","average":{"1min":14.311,"5min":18.312000000000001,"15min":18.978999999999999},"min":{"1min":9.6310000000000002,"5min":9.6310000000000002,"15min":9.6310000000000002},"max":{"1min":23.643000000000001,"5min":77.974000000000004,"15min":77.974000000000004},"last":14.619}]}]},{"osd":4,"up_from":15,"seq":64424509479,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":168785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":7.1479999999999997,"5min":9.2040000000000006,"15min":9.5459999999999994},"min":{"1min":4.234,"5min":4.234,"15min":4.234},"max":{"1min":16.420000000000002,"5min":23.677,"15min":23.677},"last":4.7679999999999998},{"interface":"front","average":{"1min":9.7490000000000006,"5min":10.183,"15min":10.255000000000001},"min":{"1min":5.0960000000000001,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":18.917999999999999,"5min":22.443999999999999,"15min":22.443999999999999},"last":14.289999999999999}]},{"osd":1,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":9.8130000000000006,"5min":11.587,"15min":11.882999999999999},"min":{"1min":5.0090000000000003,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":18.620999999999999,"5min":35.524000000000001,"15min":35.524000000000001},"last":8.2370000000000001},{"interface":"front","average":{"1min":13.038,"5min":13.85,"15min":13.984999999999999},"min":{"1min":7.2859999999999996,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":19.516999999999999,"5min":31.027000000000001,"15min":31.027000000000001},"last":15.805999999999999}]},{"osd":2,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":13.407999999999999,"5min":12.295,"15min":12.109},"min":{"1min":9.0310000000000006,"5min":8.1159999999999997,"15min":8.1159999999999997},"max":{"1min":22.754000000000001,"5min":22.754000000000001,"15min":22.754000000000001},"last":9.0990000000000002},{"interface":"front","average":{"1min":15.56,"5min":14.811999999999999,"15min":14.688000000000001},"min":{"1min":10.954000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":32.915999999999997,"5min":32.915999999999997,"15min":32.915999999999997},"last":16.614000000000001}]},{"osd":3,"last update":"Wed Dec 6 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":10.792,"5min":11.346,"15min":11.438000000000001},"min":{"1min":6.9939999999999998,"5min":6.843,"15min":6.843},"max":{"1min":17.343,"5min":25.640999999999998,"15min":25.640999999999998},"last":8.5519999999999996},{"interface":"front","average":{"1min":15.763,"5min":14.404,"15min":14.178000000000001},"min":{"1min":9.282,"5min":8.6170000000000009,"15min":8.6170000000000009},"max":{"1min":25.077999999999999,"5min":25.077999999999999,"15min":25.077999999999999},"last":13.576000000000001}]},{"osd":5,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":16.332999999999998,"5min":16.945,"15min":17.047000000000001},"min":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":36.713000000000001,"5min":50.417999999999999,"15min":50.417999999999999},"last":17.384},{"interface":"front","average":{"1min":18.265999999999998,"5min":17.788,"15min":17.707999999999998},"min":{"1min":12.552,"5min":9.9399999999999995,"15min":9.9399999999999995},"max":{"1min":26.803999999999998,"5min":48.837000000000003,"15min":48.837000000000003},"last":12.398}]},{"osd":6,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":17.388000000000002,"5min":21.440000000000001,"15min":22.116},"min":{"1min":8.7899999999999991,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":40.418999999999997,"5min":93.073999999999998,"15min":93.073999999999998},"last":18.291},{"interface":"front","average":{"1min":12.763999999999999,"5min":17.523,"15min":18.315999999999999},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":21.292000000000002,"5min":91.075999999999993,"15min":91.075999999999993},"last":10.478999999999999}]},{"osd":7,"last update":"Wed Dec 6 07:59:21 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":16.18,"15min":16.305},"min":{"1min":9.8279999999999994,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":33.988999999999997,"5min":33.988999999999997,"15min":33.988999999999997},"last":11.747},{"interface":"front","average":{"1min":19.609000000000002,"5min":17.306000000000001,"15min":16.922000000000001},"min":{"1min":14.167999999999999,"5min":12.539,"15min":12.539},"max":{"1min":32.014000000000003,"5min":32.014000000000003,"15min":32.014000000000003},"last":12.974}]}]},{"osd":6,"up_from":14,"seq":60129542181,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":168785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":8.8780000000000001,"5min":10.023999999999999,"15min":10.215},"min":{"1min":4.2779999999999996,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":15.942,"5min":70.965000000000003,"15min":70.965000000000003},"last":7.048},{"interface":"front","average":{"1min":8.6679999999999993,"5min":11.603999999999999,"15min":12.093},"min":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":15.984999999999999,"5min":73.582999999999998,"15min":73.582999999999998},"last":7.6299999999999999}]},{"osd":1,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":13.266,"5min":18.300000000000001,"15min":19.138999999999999},"min":{"1min":7.3079999999999998,"5min":4.383,"15min":4.383},"max":{"1min":39.470999999999997,"5min":217.595,"15min":217.595},"last":6.7480000000000002},{"interface":"front","average":{"1min":12.566000000000001,"5min":21.143000000000001,"15min":22.573},"min":{"1min":5.9859999999999998,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":37.878,"5min":284.96499999999997,"15min":284.96499999999997},"last":8.4540000000000006}]},{"osd":2,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":12.776999999999999,"5min":13.661,"15min":13.808},"min":{"1min":7.3120000000000003,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":19.815000000000001,"5min":46.024999999999999,"15min":46.024999999999999},"last":13.087},{"interface":"front","average":{"1min":12.541,"5min":16.094999999999999,"15min":16.687000000000001},"min":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"max":{"1min":19.689,"5min":46.966999999999999,"15min":46.966999999999999},"last":12.052}]},{"osd":3,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":14.635999999999999,"5min":15.603999999999999,"15min":15.765000000000001},"min":{"1min":9.3460000000000001,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":25.395,"5min":51.031999999999996,"15min":51.031999999999996},"last":13.744999999999999},{"interface":"front","average":{"1min":13.731,"5min":22.231000000000002,"15min":23.646999999999998},"min":{"1min":7.9880000000000004,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":21.751000000000001,"5min":189.84999999999999,"15min":189.84999999999999},"last":11.49}]},{"osd":4,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":15.287000000000001,"5min":15.763999999999999,"15min":15.843999999999999},"min":{"1min":10.183,"5min":10.183,"15min":10.183},"max":{"1min":20.337,"5min":25.013000000000002,"15min":25.013000000000002},"last":16.536999999999999},{"interface":"front","average":{"1min":15.156000000000001,"5min":14.734999999999999,"15min":14.664999999999999},"min":{"1min":10.518000000000001,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":22.789000000000001,"5min":26.831,"15min":26.831},"last":15.831}]},{"osd":5,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":19.088000000000001,"5min":18.956,"15min":18.934000000000001},"min":{"1min":10.811,"5min":10.811,"15min":10.811},"max":{"1min":43.795999999999999,"5min":50.945,"15min":50.945},"last":17.658999999999999},{"interface":"front","average":{"1min":14.279,"5min":14.353999999999999,"15min":14.366},"min":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"max":{"1min":45.25,"5min":51.948,"15min":51.948},"last":14.461}]},{"osd":7,"last update":"Wed Dec 6 07:59:19 2023","interfaces":[{"interface":"back","average":{"1min":16.399999999999999,"5min":15.888,"15min":15.803000000000001},"min":{"1min":9.1069999999999993,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":24.576000000000001,"5min":31.353999999999999,"15min":31.353999999999999},"last":23.699000000000002},{"interface":"front","average":{"1min":17.803999999999998,"5min":19.163,"15min":19.388999999999999},"min":{"1min":12.43,"5min":12.43,"15min":12.43},"max":{"1min":30.562000000000001,"5min":30.562000000000001,"15min":30.562000000000001},"last":25.161999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542182,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":843776,"data_stored":562179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":8.0700000000000003,"5min":8.923,"15min":9.0649999999999995},"min":{"1min":5.1520000000000001,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":15.169,"5min":23.09,"15min":23.09},"last":5.3369999999999997},{"interface":"front","average":{"1min":8.5649999999999995,"5min":9.4199999999999999,"15min":9.5619999999999994},"min":{"1min":4.7930000000000001,"5min":4.7930000000000001,"15min":4.7930000000000001},"max":{"1min":19.276,"5min":20.728000000000002,"15min":20.728000000000002},"last":6.8029999999999999}]},{"osd":1,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":11.742000000000001,"5min":13.407999999999999,"15min":13.686},"min":{"1min":4.8979999999999997,"5min":4.8979999999999997,"15min":4.8979999999999997},"max":{"1min":24.928000000000001,"5min":67.347999999999999,"15min":67.347999999999999},"last":12.474},{"interface":"front","average":{"1min":13.577,"5min":12.922000000000001,"15min":12.813000000000001},"min":{"1min":6.2640000000000002,"5min":5.8090000000000002,"15min":5.8090000000000002},"max":{"1min":23.263999999999999,"5min":39.052,"15min":39.052},"last":8.3829999999999991}]},{"osd":2,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.504,"5min":12.599,"15min":12.281000000000001},"min":{"1min":7.1639999999999997,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":31.873000000000001,"5min":31.873000000000001,"15min":31.873000000000001},"last":15.087},{"interface":"front","average":{"1min":14.506,"5min":15.742000000000001,"15min":15.948},"min":{"1min":6.8209999999999997,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":30.876000000000001,"5min":34.585000000000001,"15min":34.585000000000001},"last":10.662000000000001}]},{"osd":3,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":15.291,"5min":15.486000000000001,"15min":15.518000000000001},"min":{"1min":9.5589999999999993,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":28.518000000000001,"5min":30.355,"15min":30.355},"last":14.089},{"interface":"front","average":{"1min":15.138999999999999,"5min":15.185,"15min":15.193},"min":{"1min":7.9359999999999999,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":31.792000000000002,"5min":31.792000000000002,"15min":31.792000000000002},"last":16.748000000000001}]},{"osd":4,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":13.646000000000001,"5min":26.282,"15min":28.388000000000002},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":18.556999999999999,"5min":334.86500000000001,"15min":334.86500000000001},"last":15.987},{"interface":"front","average":{"1min":14.972,"5min":26.417000000000002,"15min":28.324999999999999},"min":{"1min":10.587,"5min":7.431,"15min":7.431},"max":{"1min":23.646999999999998,"5min":331.57900000000001,"15min":331.57900000000001},"last":9.6639999999999997}]},{"osd":6,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":15.94,"5min":19.248000000000001,"15min":19.798999999999999},"min":{"1min":9.6400000000000006,"5min":9.6400000000000006,"15min":9.6400000000000006},"max":{"1min":27.963999999999999,"5min":46.034999999999997,"15min":46.034999999999997},"last":18.097999999999999},{"interface":"front","average":{"1min":16.256,"5min":18.359999999999999,"15min":18.710000000000001},"min":{"1min":9.6709999999999994,"5min":9.6709999999999994,"15min":9.6709999999999994},"max":{"1min":25.742000000000001,"5min":35.680999999999997,"15min":35.680999999999997},"last":11.242000000000001}]},{"osd":7,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.593,"5min":27.978999999999999,"15min":30.210000000000001},"min":{"1min":9.1329999999999991,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":26.213000000000001,"5min":332.322,"15min":332.322},"last":17.366},{"interface":"front","average":{"1min":16.748999999999999,"5min":30.989000000000001,"15min":33.363},"min":{"1min":11.039,"5min":11.039,"15min":11.039},"max":{"1min":21.757000000000001,"5min":325.93599999999998,"15min":325.93599999999998},"last":13.089}]}]},{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":159872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":9.2400000000000002,"5min":9.4960000000000004,"15min":9.5380000000000003},"min":{"1min":6.1849999999999996,"5min":5.0449999999999999,"15min":5.0449999999999999},"max":{"1min":15.425000000000001,"5min":20.558,"15min":20.558},"last":8.3879999999999999},{"interface":"front","average":{"1min":12.69,"5min":12.125,"15min":12.031000000000001},"min":{"1min":8.2319999999999993,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":16.902000000000001,"5min":20.893999999999998,"15min":20.893999999999998},"last":16.981000000000002}]},{"osd":1,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":12.448,"5min":15.959,"15min":16.544},"min":{"1min":7.7889999999999997,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":20.911000000000001,"5min":51.890000000000001,"15min":51.890000000000001},"last":10.426},{"interface":"front","average":{"1min":8.5869999999999997,"5min":8.4309999999999992,"15min":8.4049999999999994},"min":{"1min":4.7519999999999998,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":21.609999999999999,"5min":35.075000000000003,"15min":35.075000000000003},"last":13.339}]},{"osd":2,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":13.022,"5min":12.69,"15min":12.635},"min":{"1min":6.6440000000000001,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":30.649999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":9.7309999999999999},{"interface":"front","average":{"1min":12.874000000000001,"5min":11.499000000000001,"15min":11.27},"min":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":23.489999999999998,"5min":23.489999999999998,"15min":23.489999999999998},"last":12.054}]},{"osd":3,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":27.213999999999999,"15min":29.18},"min":{"1min":7.9829999999999997,"5min":7.3140000000000001,"15min":7.3140000000000001},"max":{"1min":32.116,"5min":228.17099999999999,"15min":228.17099999999999},"last":9.0860000000000003},{"interface":"front","average":{"1min":15.337999999999999,"5min":29.597000000000001,"15min":31.972999999999999},"min":{"1min":10.156000000000001,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":26.178000000000001,"5min":270.13,"15min":270.13},"last":15.888}]},{"osd":4,"last update":"Wed Dec 6 07:59:22 2023","interfaces":[{"interface":"back","average":{"1min":14.018000000000001,"5min":16.728000000000002,"15min":17.18},"min":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":25.247,"5min":86.980999999999995,"15min":86.980999999999995},"last":7.6340000000000003},{"interface":"front","average":{"1min":17.876000000000001,"5min":20.236000000000001,"15min":20.629000000000001},"min":{"1min":9.2059999999999995,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":31.477,"5min":99.021000000000001,"15min":99.021000000000001},"last":7.9770000000000003}]},{"osd":5,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":20.492000000000001,"5min":21.867999999999999,"15min":22.097999999999999},"min":{"1min":10.611000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"max":{"1min":48.439,"5min":97.191000000000003,"15min":97.191000000000003},"last":15.271000000000001},{"interface":"front","average":{"1min":18.427,"5min":20.876999999999999,"15min":21.285},"min":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":49.709000000000003,"5min":64.048000000000002,"15min":64.048000000000002},"last":14.487}]},{"osd":6,"last update":"Wed Dec 6 07:59:20 2023","interfaces":[{"interface":"back","average":{"1min":14.638999999999999,"5min":17.018999999999998,"15min":17.416},"min":{"1min":9.3729999999999993,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":19.253,"5min":74.064999999999998,"15min":74.064999999999998},"last":17.686},{"interface":"front","average":{"1min":19.581,"5min":22.021000000000001,"15min":22.427},"min":{"1min":11.448,"5min":11.448,"15min":11.448},"max":{"1min":34.610999999999997,"5min":92.477999999999994,"15min":92.477999999999994},"last":12.738}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:00:15.751 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:15.749+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:15.850 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:15.850 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:15.851 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:15.850+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:15.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:15.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:00:15.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:15.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:15.952 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:15.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:16.011 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.011 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:16.011 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:16.011 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:16.023 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:16.024 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:16.054 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:16.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:16.154 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:16.153+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:16.155 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.155 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:16.155 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:16.155 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:16.155 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:16.255 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:16.255+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:16.298 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:16.298 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:16.392 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:16.392+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:16.457 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:16.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:16.479 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.479 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:16.479 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:16.480 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:16.558 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:16.557+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:16.660 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:16.659+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:16.667 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.668 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:16.668 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:16.668 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:16.668 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:16.760 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:16.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:16.861 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:16.861+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:16.863 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:16.863 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:16.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:16.863 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:16.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 176.73175473000001, 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:16.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:16.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:16.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:16.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.872 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 175.26064127699999, 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:16.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:16.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:16.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:16.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 175.030448898, 2023-12-06T08:00:16.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.880 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:16.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:16.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:16.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 175.02974134900001, 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:16.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:16.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:16.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 175.02910435499999, 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:16.891 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:16.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.97203916000001, 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.897 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.902 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.97054838899999, 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.904 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:16.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.904 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:16.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.93819872700001, 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.910 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.911 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.911 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:16.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.911 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:16.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.916 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.93156795199999, 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:16.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:16.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:16.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:16.922 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.930917271, 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.92744458000001, 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:16.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:16.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:16.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.934 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.92630585800001, 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:16.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:16.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.92535816099999, 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:16.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:16.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:16.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.896994167, 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.946 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:16.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:16.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:16.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:16.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:16.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.86878465000001, 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:16.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:16.960 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.258586716, 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.25794187599999, 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.966 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:16.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:16.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:16.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:16.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.23747512200001, 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:16.971 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:16.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:16.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.976 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.23672620599999, 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.977 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:16.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.982 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.23608413700001, 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:16.983 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:16.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:16.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:16.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:16.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:16.988 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:16.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:16.990 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:16.990+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:17.091 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:17.091+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:17.162 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.163 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:17.163 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:17.163 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:17.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:17.213 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:17.213+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:17.293 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:17.292+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:17.347 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.348 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:17.348 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:17.348 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:17.348 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:17.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:17.396 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:00:17.396 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:17.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:17.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:17.495 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:17.495+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:17.502 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.502 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:17.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:17.502 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:17.502 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:17.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:17.596 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:17.595+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:17.685 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.685 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:17.685 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:17.685 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:17.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:17.696 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:17.696+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:17.797 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:17.797+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:17.827 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.827 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:17.827 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:17.827 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:17.827 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:17.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:17.899 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:17.898+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:17.995 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:17.995 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:17.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:17.995 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 158.578268847, 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:17.999 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:18.004 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:18.001+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:18.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:18.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:00:18.102 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:18.101+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:18.138 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:18.139 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:18.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:18.163 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:18.234 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:18.233+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:18.303 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:18.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:18.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:18.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:18.317 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:18.317 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:18.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:18.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:18.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:18.409 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:18.409+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:18.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:18.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:18.506 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:18.506+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:18.512 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:18.512 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:18.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:18.573 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.371 105999== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:18.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:18.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:18.607 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:18.607+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:18.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:18.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:18.705 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:18.705 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:18.705 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:18.705 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:18.709 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:18.708+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:18.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:18.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:18.764 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:18.816 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.614 105999== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:18.851 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:18.851 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:18.851 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:18.850+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:18.883 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:18.884 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:18.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:18.909 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:18.909+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:18.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:18.940 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:19.010 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:19.010+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:19.021 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.021 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:19.111 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:19.110+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:19.158 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.158 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:19.211 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:19.211+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:19.312 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:19.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:19.334 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.335 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:19.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:00:19.414 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:19.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:19.473 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.473 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:19.514 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:19.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:19.614 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:19.614+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:19.650 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.651 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:19.716 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:19.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:19.790 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.790 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:19.819 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:19.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:19.920 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:19.920+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:19.928 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:19.928 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:19.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:20.022 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:20.022+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:20.073 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.074 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:20.124 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:20.124+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:20.227 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:20.226+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:20.257 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.257 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:20.327 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:20.327+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:20.398 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.398 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:20.451 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:20.451+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:20.530 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:20.530+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:20.544 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.544 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:20.638 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:20.638+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:20.723 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.723 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.732 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:20.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:20.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:20.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:00:20.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:00:20.753 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-06T08:00:20.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-06T08:00:20.833 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:20.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:20.906 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:20.907 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:20.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:20.934 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:20.933+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:20.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:00:21.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-06T08:00:21.035 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:21.035+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:21.122 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:21.122 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:21.125 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:00:21.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:21.136 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:21.136+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:21.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:21.243 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:21.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:21.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:21.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:21.320 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:21.320 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:21.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-06T08:00:21.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:21.337 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:21.337+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:21.379 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:21.380 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:21.438 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:21.438+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:21.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:21.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:21.529 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:21.529 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:21.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:21.530 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:21.530 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:21.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:21.540 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:21.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:21.625 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:21.625 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:21.640 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:21.640+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:21.713 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:21.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:21.727 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:21.728 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:21.728 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:21.728 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:21.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:21.746 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:21.746+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:21.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:21.806 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:21.843 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:21.842+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:21.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-06T08:00:21.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:21.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:21.934 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:21.934 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:21.934 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:21.934 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:21.934 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:21.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:21.944 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:21.943+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:21.953 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:22.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:22.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:22.044 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:22.044+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.116 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 162.69849958399999, 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:22.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:22.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:22.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:22.121 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:22.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:22.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2023-12-06T08:00:22.171 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:22.170+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:22.247 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:22.246+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:22.255 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:22.256 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:22.256 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:22.256 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:22.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:22.265 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.062 105999== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:22.318 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.116 105999== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:22.349 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:22.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:22.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2023-12-06T08:00:22.436 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:22.436 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:22.436 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:22.436 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:22.436 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:22.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:22.491 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:22.486+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:22.549 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:22.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:22.560 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.358 105999== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:22.618 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:22.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:22.618 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:22.619 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:22.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:22.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2023-12-06T08:00:22.685 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:22.685+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:22.752 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:22.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:22.802 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:22.802 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:22.803 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:22.803 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:22.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:22.852 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:22.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:22.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2023-12-06T08:00:22.941 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:22.941 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:22.941 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:22.941 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:22.941 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:22.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:22.963 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:22.961+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:23.066 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:23.066+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 182.14030297299999, 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.127 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:23.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:23.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:23.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.131 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.476521972, 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:23.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:23.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.47324134300001, 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:23.137 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:23.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.16846472700001, 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.141 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:23.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.145 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.16762163000001, 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.14508612399999, 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:23.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:23.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:23.153 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.14309370500001, 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:23.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:23.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.14256144000001, 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:23.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:23.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:23.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.162 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.125636531, 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 181.124799897, 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.10035262900001, 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:23.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:23.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.175 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.09445026500001, 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:23.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:23.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:23.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.08974426899999, 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:23.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.184 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.08744239399999, 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:23.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.086944071, 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.08649016000001, 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.08526775999999, 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:23.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:23.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:23.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 174.440101779, 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 174.43266336400001, 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 174.42258999200001, 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:23.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:23.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:23.213 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:23.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:00:23.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:23.221 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:23.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:23.316 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:23.315+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:23.394 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:23.394 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:23.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:23.417 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:23.417+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:23.517 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:23.517+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:23.593 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:23.593 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:23.593 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:23.593 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:23.593 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:23.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:23.619 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:23.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:23.719 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:23.719+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:23.737 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:23.737 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:23.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:23.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:23.737 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:23.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:23.830 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:23.829+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:23.918 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:23.918 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:23.921 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:23.921+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:23.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:24.023 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:24.023+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:24.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:00:24.094 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:24.094 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:24.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:24.124 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:24.124+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:24.227 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:24.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 163.69772190699999, 2023-12-06T08:00:24.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:24.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:24.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:24.278 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:24.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:24.331 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:24.331+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:24.416 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:24.416 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:24.417 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:24.417 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:24.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:24.433 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:24.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:24.538 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:24.534+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:24.600 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:24.601 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:24.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:24.635 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:24.634+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:24.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:24.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:24.692 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:00:24.738 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:24.736+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:24.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:24.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:24.801 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:24.801 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:24.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:24.837 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:24.837+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:24.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:24.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:24.965 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:24.965 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:24.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:24.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:24.990 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:24.990+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:24.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:24.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:25.039 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:25.039+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:25.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:25.140 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.140 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:25.140 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:25.140+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:25.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:25.170 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:25.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:25.242 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:25.241+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:25.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:25.271 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:25.292 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.293 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:25.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:25.343 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:25.342+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:25.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:25.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:25.444 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:25.443+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:25.485 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.485 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:25.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:25.545 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:25.544+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:25.627 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.627 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:25.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:25.680 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:25.679+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:25.686 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:00:25.702 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T08:00:25.702 INFO:tasks.mon_recovery:stopping all monitors 2023-12-06T08:00:25.702 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T08:00:25.702 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:00:25.728 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:00:25.715+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 97687) UID: 0 2023-12-06T08:00:25.733 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:00:25.717+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T08:00:25.748 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:25.747+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:25.784 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.784 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:25.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:25.879 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:25.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:25.949 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:25.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:25.996 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:25.996 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:26.085 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:26.084+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:26.151 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:26.151+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:26.176 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.177 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:26.203 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:34.836 97689== 2023-12-06T08:00:26.203 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:34.836 97689== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:00:26.213 INFO:tasks.ceph.mon.b.smithi107.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:00:26.262 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:26.261+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:26.311 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.311 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:26.353 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:26.353+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:26.449 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.454 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:26.453+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:26.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:26.555 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:26.554+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:26.585 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.585 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:26.656 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:26.655+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:26.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:00:26.723 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.723 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:26.757 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:26.756+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:26.859 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:26.858+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:26.864 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:26.864 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:26.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:26.961 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:26.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:27.043 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.043 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:27.061 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:27.060+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:27.162 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:27.162+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:27.180 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.180 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:27.264 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:27.262+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.364 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:27.364+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:27.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:27.373 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:00:27.465 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:27.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:27.542 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.542 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:27.567 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:27.566+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:27.668 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:27.668+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:27.774 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:27.774+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:27.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:00:27.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:00:27.859 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:27.859 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:27.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:27.909 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:00:27.909 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:00:27.912 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:27.910+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:27.971 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:27.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:27.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:00:27.994 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:00:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:28.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:28.072 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:28.071+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:28.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:00:28.080 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:00:28.169 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:00:28.169 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:00:28.173 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:28.172+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:28.219 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:00:28.219 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:00:28.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:28.257 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:00:28.257 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:00:28.274 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:28.273+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:28.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:00:28.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:00:28.364 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:28.364 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:28.364 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:28.364 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:28.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:28.415 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:28.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:28.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:00:28.430 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:00:28.483 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:28.482+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:28.542 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:28.543 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:28.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:28.577 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:28.577+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:28.678 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:28.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:28.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:00:28.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:00:28.719 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:28.720 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:28.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:28.720 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:28.720 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:28.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:28.779 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:28.779+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:28.854 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:28.855 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:28.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:28.880 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:28.879+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:28.981 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:28.979+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:28.991 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:28.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:28.991 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:28.991 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:28.991 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:29.082 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:29.082+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.225 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.09447431800001, 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:29.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:29.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:29.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:29.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:29.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:29.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:29.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.623360865, 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.236 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:29.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.39316848600001, 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.242 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:29.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:29.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:29.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:29.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.39246093700001, 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.248 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:29.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.39182394299999, 2023-12-06T08:00:29.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:29.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:29.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:29.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.267 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.33475874800001, 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:29.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:29.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:29.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:29.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.273 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.33326797699999, 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:29.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:29.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:29.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:29.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.30091831499999, 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:29.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:29.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:29.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:29.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:29.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.284 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.29428754, 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:29.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:29.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.289 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.293636859, 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.290 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:29.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:29.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 186.29016416799999, 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:29.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:29.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:29.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:29.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 186.28902544600001, 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:29.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:29.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:29.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 186.288077749, 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:29.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.314 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 186.25971375500001, 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:29.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:29.320 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 186.23150423800001, 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:29.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.621306304, 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:29.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:29.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:29.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:29.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:29.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:29.333 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.62066146399999, 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:29.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.60019471000001, 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:29.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.59944579399999, 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.344 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:29.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.348 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:29.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.59880372500001, 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:29.351 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:29.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:29.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:29.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:29.356 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:29.359 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:29.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:29.459 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:29.458+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:29.484 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:29.484 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:29.484 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:29.484 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:29.560 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:29.558+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:29.626 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:29.627 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:29.677 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:29.677+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:29.761 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:29.760+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:29.805 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:29.806 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:29.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:29.806 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:29.806 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:29.898 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:29.897+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:29.963 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:29.963+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:29.982 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:29.983 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:29.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:30.075 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:30.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:30.160 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:30.160 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:30.165 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:30.165+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:30.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:30.266 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:30.265+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 170.920467758, 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:30.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:30.342 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:30.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:30.387 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:30.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:30.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:00:30.486 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:30.486 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:30.486 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:30.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:30.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:30.570 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:30.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:30.624 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:30.624 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:30.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:30.670 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:30.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:30.765 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:30.765 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:30.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:30.765 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:30.766 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:30.772 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:30.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:30.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:30.873 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:30.873+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:30.942 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:30.943 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:30.943 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:30.943 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:30.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:30.974 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:30.974+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:31.075 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:31.075+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:31.121 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:31.122 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:31.122 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:31.122 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:31.122 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:31.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:31.176 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:31.176+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:31.277 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:31.277+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:31.298 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 190.31787405399999, 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.654093053, 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:31.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.65081242400001, 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:31.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:31.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.34603580800001, 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:31.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:31.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.319 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.34519271100001, 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:31.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:31.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:31.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.32265720500001, 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:31.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:31.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.32066478600001, 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:31.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:31.334 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.320132521, 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.30320761199999, 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:31.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 189.302370978, 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:31.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.27792371000001, 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:31.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:31.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.272021346, 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:31.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.26731534999999, 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:31.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.26501347499999, 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:31.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.264515152, 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:31.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:31.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:31.366 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.26406124100001, 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.26283884099999, 2023-12-06T08:00:31.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:31.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 182.61767286, 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:31.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 182.610234445, 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:31.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 182.60016107300001, 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:31.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:31.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.386 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:31.387 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:31.387 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:31.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:31.389 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:31.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:31.489 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:31.489+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:31.557 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:31.557 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:31.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:31.610 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:31.609+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:31.691 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:31.690+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:31.736 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:31.737 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:31.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:31.792 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:31.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:00:31.804 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T08:00:31.804 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T08:00:31.804 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T08:00:31.804 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:00:31.820 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:00:31.808+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 104673) UID: 0 2023-12-06T08:00:31.820 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:00:31.809+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T08:00:31.876 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:31.876 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:31.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:31.876 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:31.876 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:31.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:31.893 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:31.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:31.994 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:31.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:32.020 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:32.020 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:32.020 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:32.020 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:32.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:32.095 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:32.094+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:32.196 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:32.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:32.205 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:32.205 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:32.206 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:32.206 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:32.206 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:32.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:32.297 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:32.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:32.327 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:40.948 104678== 2023-12-06T08:00:32.327 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:40.948 104678== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:00:32.338 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:32.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 171.770752028, 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:32.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:32.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:32.351 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:32.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:32.430 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:32.430+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:32.499 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:32.498+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:32.514 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:32.514 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:32.515 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:32.515 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:32.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:32.599 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:32.599+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:32.693 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:32.694 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:32.700 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:32.699+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:32.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:32.800 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:32.799+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:32.838 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:32.839 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:32.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:32.839 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:32.839 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:32.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:32.901 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:32.901+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:33.002 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:33.002+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:33.024 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:33.024 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:33.024 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:33.024 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:33.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:33.121 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:33.121+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:33.206 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:33.206 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:33.206 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:33.206 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:33.206+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:33.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:33.307 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:33.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:33.408 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:33.408+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:33.436 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.30742858900001, 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:33.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:33.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:33.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.836315136, 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.444 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:33.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.60612275700001, 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.450 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.60541520800001, 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.60477821399999, 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.54771301900001, 2023-12-06T08:00:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.54622224799999, 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.51387258599999, 2023-12-06T08:00:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.507241811, 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.50659113, 2023-12-06T08:00:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.50311843899999, 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.50197971700001, 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.50103202, 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.47266802600001, 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.44445850899999, 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 184.834260575, 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 184.833615735, 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 184.81314898100001, 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 184.81240006499999, 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 184.81175799600001, 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:33.559 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:33.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:33.561 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:33.561+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:33.662 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:33.662+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:33.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:00:33.688 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:00:33.688 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.18910569408161548 2023-12-06T08:00:33.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 0.18910569408161548 2023-12-06T08:00:33.731 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:33.732 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:33.732 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:33.732 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:33.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:33.764 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:33.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:33.864 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:33.864+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:33.933 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:33.933 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:33.933 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:33.933 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:33.933 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:33.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:33.965 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:33.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:34.066 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:34.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:34.093 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:34.094 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:34.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:34.094 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:34.094 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:34.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:34.167 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:34.167+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:34.235 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:34.235 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:34.235 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:34.235 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:34.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:34.268 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:34.268+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:34.369 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:34.369+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:34.373 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:34.373 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:34.373 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:34.373 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:34.373 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:34.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:34.470 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:34.470+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 175.13574181300001, 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:34.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:34.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:34.557 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:34.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:34.609 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:34.609+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:34.671 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:34.671+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:34.707 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:34.707 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:34.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:34.802 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:34.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:34.872 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:34.872+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:34.883 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:34.883 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:34.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:34.977 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:34.977+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:35.062 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:35.062 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:35.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:35.063 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:35.063 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:35.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:35.114 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:35.113+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:35.175 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:35.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:35.240 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:35.240 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:35.240 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:35.241 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:35.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:35.276 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:35.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:35.376 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:35.376+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:35.418 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:35.419 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:35.419 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:35.419 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:35.419 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:35.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:35.477 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:35.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:35.598 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 194.61856277199999, 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.954781771, 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.95150114200001, 2023-12-06T08:00:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.64672452600001, 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.64588142900001, 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:35.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.62334592299999, 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.624 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:35.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.630 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.62135350400001, 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:35.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.62082123900001, 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:35.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:35.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:35.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.60389633, 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.643 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:35.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:35.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:35.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.603059696, 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:35.648 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:35.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:35.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:35.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.57861242800001, 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.57271006400001, 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:35.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:35.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.56800406799999, 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:35.664 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:35.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:35.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:35.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:35.670 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.56570219299999, 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:35.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:35.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:35.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:35.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.56520387, 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:35.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:35.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:35.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.56474995900001, 2023-12-06T08:00:35.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:35.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:35.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:35.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 192.56352755899999, 2023-12-06T08:00:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:35.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:35.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 186.918361578, 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:35.702 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 186.91092316300001, 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:35.706 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 186.90084979100001, 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:35.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:35.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:35.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:35.711 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:35.712 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:35.598+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:35.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:35.754 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:35.753+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:35.815 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:35.815+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:35.883 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:35.884 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:35.884 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:35.884 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:35.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:35.939 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:35.938+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:36.017 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:36.016+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:36.027 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:36.027 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:36.028 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:36.028 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:36.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:36.120 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:36.120+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:36.207 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:36.207 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:36.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:36.207 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:36.207 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:36.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:36.259 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:36.258+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:36.320 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:36.319+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:36.385 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:36.385 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:36.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:36.440 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:36.439+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:36.521 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:36.521+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:36.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:36.623 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:36.622+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:36.702 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 176.12625223699999, 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:36.703 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:36.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:36.707 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:36.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:36.756 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:36.756+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:36.825 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:36.824+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:36.849 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:36.849 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:36.849 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:36.849 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:36.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:36.948 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:36.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:37.026 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:37.026+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:37.030 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:37.031 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:37.031 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:37.128 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:37.127+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:37.207 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:37.208 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:37.262 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:37.260+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:37.329 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:37.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:37.342 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.343 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:37.343 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:37.343 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:37.430 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:37.429+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:37.523 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.523 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:37.524 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:37.524 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:37.524 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.531 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:37.531+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:37.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:37.632 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:37.632+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.722 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.59170320300001, 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.728 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 196.12058974999999, 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:37.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:37.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.89039737100001, 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.734 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:37.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:37.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:37.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:37.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:37.742 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.88968982200001, 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.88905282799999, 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:37.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.83198763300001, 2023-12-06T08:00:37.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.755 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:37.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.83049686199999, 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.760 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.79814719999999, 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:37.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:37.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:37.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.791516425, 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:37.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:37.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.790865744, 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.779 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:37.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:37.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:37.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 194.78739305299999, 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.784 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 194.78625433100001, 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:37.789 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:37.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:37.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 194.78530663399999, 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:37.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:37.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 194.75694264000001, 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:37.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:37.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 194.72873312300001, 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:37.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:37.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.118535189, 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:37.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:37.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:37.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.11789034899999, 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.815 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:37.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:37.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:37.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:37.820 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.09742359500001, 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:37.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.825 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.09667467899999, 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.09603261000001, 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:37.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:37.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:37.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:37.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.836 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:37.836 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:37.836 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:37.840 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:37.838+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:00:37.906 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T08:00:37.907 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T08:00:37.907 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T08:00:37.907 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:00:37.921 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:00:37.911+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 105994) UID: 0 2023-12-06T08:00:37.922 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:00:37.913+0000 9f54640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2023-12-06T08:00:37.939 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:37.938+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:37.984 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:37.985 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:37.985 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:37.985 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:37.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:38.039 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:38.039+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:38.140 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:38.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:38.164 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:38.164 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:38.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:38.240 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:38.240+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:38.304 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:38.304 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:38.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:38.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:38.304 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:38.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:38.341 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:38.341+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:38.442 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:38.442 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:38.442 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:38.442 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:38.442 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:38.442+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:38.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:38.455 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:28.253 105999== 2023-12-06T08:00:38.455 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:28.253 105999== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:00:38.466 INFO:tasks.ceph.mon.a.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:00:38.544 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:38.543+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:38.584 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:38.584 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:38.584 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:38.585 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:38.585 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:38.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:38.679 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:38.679+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:38.755 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:38.755+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:38.770 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 179.353814223, 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:38.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:38.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:38.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:38.775 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:38.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:38.846 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:38.845+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:38.924 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:38.924 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:38.924 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:38.924 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:38.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:38.979 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:38.978+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:39.050 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:39.049+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:39.062 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.062 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:39.062 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:39.062 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:39.062 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:39.150 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:39.150+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:39.239 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:39.240 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:39.252 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:39.251+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:39.352 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:39.352+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:39.381 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.382 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:39.382 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:39.382 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:39.453 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:39.453+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:39.517 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.517 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:39.518 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:39.518 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:39.518 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:39.573 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:39.572+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:39.655 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:39.655+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:39.697 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.697 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:39.698 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 198.716403654, 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:39.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:39.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:39.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:39.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:39.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:39.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 198.05262265299999, 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:39.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 198.049342024, 2023-12-06T08:00:39.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:39.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:39.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.744565408, 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:39.715 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:39.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:39.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.743722311, 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:39.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:39.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:39.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.721186805, 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:39.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.719194386, 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:39.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.71866212099999, 2023-12-06T08:00:39.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:39.740 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.70173721200001, 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.70090057799999, 2023-12-06T08:00:39.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:39.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:39.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.748 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.67645331, 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:39.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.752 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.67055094599999, 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:39.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:39.756 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.66584495000001, 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:39.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.66354307500001, 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:39.764 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.66304475199999, 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.662590841, 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:39.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:39.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 196.66136844100001, 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 191.01620245999999, 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:39.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:39.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 191.00876404499999, 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:39.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:39.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 190.998690673, 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:39.788 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:39.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:39.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:39.793 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:39.795 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:39.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:39.895 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:39.895+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:39.964 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:39.964 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:39.964 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:39.964 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:39.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:39.996 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:39.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:40.097 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:40.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:40.141 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:40.142 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:40.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:40.199 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:40.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:40.282 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:40.283 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:40.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:40.283 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:40.283 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:40.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:40.300 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:40.300+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:40.406 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:40.405+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:40.420 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:40.420 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:40.420 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:40.420 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:40.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:40.514 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:40.513+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:40.597 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:40.597 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:40.597 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:40.597 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:40.597 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:40.603 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:40.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:40.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:40.704 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:40.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 180.21584006099999, 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:40.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:40.796 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:40.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:40.804 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:40.804+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:40.906 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:40.905+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:40.936 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:40.937 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:40.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:41.006 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:41.006+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:41.073 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:41.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:41.073 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:41.074 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:41.074 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:41.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:41.124 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:41.124+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:41.208 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:41.207+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:41.253 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:41.253 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:41.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:41.253 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:41.253 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:41.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:41.309 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:41.308+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:41.390 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:41.390 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:41.390 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:41.390 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:41.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:41.443 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:41.443+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:41.511 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:41.511+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:41.530 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:41.530 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:41.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:41.612 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:41.612+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:41.712 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:41.712+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:41.764 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.635362187, 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.765 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:41.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.770 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 200.16424873400001, 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:41.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:41.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:41.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:41.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.934056355, 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.780 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:41.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:41.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:41.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:41.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.933348806, 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.785 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:41.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:41.790 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.93271181200001, 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:41.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:41.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:41.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:41.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.875646617, 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:41.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.87415584600001, 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.801 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:41.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:41.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:41.806 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.84180618400001, 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.807 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:41.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:41.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:41.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:41.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.83517540899999, 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.812 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.83452472799999, 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:41.817 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:41.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.822 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.83105203700001, 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.829913315, 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.828 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:41.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:41.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.833 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.82896561800001, 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.834 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:41.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.800601624, 2023-12-06T08:00:41.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.772392107, 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:41.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.16219417299999, 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.850 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:41.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:41.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.16154933300001, 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.855 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:41.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:41.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:41.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:41.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.860 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.141082579, 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:41.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.14033366300001, 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:41.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:41.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:41.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 193.139691594, 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:41.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:41.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:41.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:41.876 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:41.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:41.878 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:41.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:41.995 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:41.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:42.009 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:42.009 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:42.009 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:42.080 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:42.079+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:42.181 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:42.180+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:42.185 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.185 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:42.186 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:42.186 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:42.186 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:42.281 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:42.281+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:42.361 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.361 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:42.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:42.361 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:42.361 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:42.411 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:42.411+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:42.483 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:42.483+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:42.538 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:42.538 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:42.589 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:42.589+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:42.685 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:42.685+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:42.714 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.714 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:42.714 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:42.714 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:42.714 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:42.786 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:42.786+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 183.43504211499999, 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:42.854 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:42.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:42.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 3.27154335, 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:42.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:00:42.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:42.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:42.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:42.863 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:42.863 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:42.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:42.909 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:42.908+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:42.990 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:42.990+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:43.004 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.005 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:43.005 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:43.005 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:43.093 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:43.091+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:43.183 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:43.184 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.193 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:43.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:43.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:43.293 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:43.293+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:43.322 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:43.323 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:43.394 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:43.393+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:43.468 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:43.468 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:43.495 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:43.495+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:43.596 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:43.595+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:43.650 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:43.650 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:43.650 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:43.650 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:43.697 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:43.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:43.799 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:43.799+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 202.89040063100001, 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:43.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 202.22661962999999, 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:43.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:43.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:43.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 202.223339001, 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:43.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.918562385, 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.917719288, 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.890 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:43.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:43.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:43.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.895183782, 2023-12-06T08:00:43.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:43.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:43.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.893191363, 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:43.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.909 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:43.909 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:43.909 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.892659098, 2023-12-06T08:00:43.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:43.910 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.910 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.910 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.914 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.87573418900001, 2023-12-06T08:00:43.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:43.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:43.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:43.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:43.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:43.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.926 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.87489755499999, 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.927 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:43.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:43.932 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.850450287, 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:43.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:43.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.84454792299999, 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:43.938 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:43.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.83984192700001, 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.83754005200001, 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.947 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:43.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.83704172899999, 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.836587818, 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.956 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:43.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:43.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:43.960 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.83536541800001, 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:43.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:43.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.964 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 195.19019943699999, 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.968 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 195.18276102199999, 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 195.17268765, 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:43.973 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:43.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:43.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:43.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:43.980 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:43.979+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:00:44.009 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T08:00:44.009 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T08:00:44.009 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2023-12-06T08:00:44.009 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T08:00:44.010 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T08:00:44.022 INFO:tasks.ceph.mon.b:Started 2023-12-06T08:00:44.023 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T08:00:44.023 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T08:00:44.025 INFO:tasks.ceph.mon.c:Started 2023-12-06T08:00:44.025 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T08:00:44.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:00:44.079 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:44.079+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:44.152 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.152 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:44.152 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:44.152 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:44.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:44.208 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:44.207+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:44.285 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:44.284+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:44.294 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.294 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:44.294 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:44.294 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:44.294 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:44.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:44.390 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:44.389+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:44.476 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.476 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:44.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:44.476 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:44.476 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:44.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:44.487 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:44.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:44.587 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:44.587+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:44.658 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.658 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:44.658 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:44.658 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:44.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:44.688 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:44.688+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:44.788 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:44.788+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:44.798 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.798 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:44.798 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:44.798 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:44.798 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:44.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:44.892 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:44.891+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:44.982 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:44.982 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:44.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:44.982 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 184.40317241299999, 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:44.983 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:44.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:44.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:44.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:44.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:44.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:44.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:44.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:44.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:44.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:44.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:44.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:44.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:44.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:44.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:44.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:44.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:44.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:00:44.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.285995067, 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.994 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:44.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:00:44.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:00:44.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:44.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:44.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:44.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:00:45.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:00:45.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:45.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:45.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:45.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:00:45.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:00:45.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:45.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:45.002 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:45.003 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:45.009 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:45.004+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:45.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:45.106 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:45.105+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:45.138 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:45.139 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:45.139 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:45.139 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:45.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:45.207 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:45.206+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:45.275 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:45.275 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:45.275 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:45.275 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:45.275 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:45.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:45.336 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:45.330+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:45.409 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:45.408+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:45.418 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:45.418 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:45.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:45.419 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:45.419 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:45.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:45.509 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:45.509+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:45.605 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:45.606 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:45.606 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:45.606 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:45.610 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:45.610+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:45.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:45.711 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:45.711+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:45.790 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:45.790 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:45.790 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:45.790 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:45.790 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:45.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:45.840 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:45.840+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:45.913 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:45.912+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:46.029 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 205.896955496, 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:46.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:46.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:46.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:46.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:46.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:46.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:46.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.038 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.039 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.42584204299999, 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.040 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:46.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:46.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:46.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:46.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.195649664, 2023-12-06T08:00:46.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:46.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:46.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:46.053 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.194942115, 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.054 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:46.055 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:46.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:46.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:46.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:46.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:46.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:46.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.19430512100001, 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.067 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:46.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:46.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:46.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:46.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:46.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:46.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.078 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.13723992600001, 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.079 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:46.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:46.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:46.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.13574915500001, 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.10339949300001, 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.094 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.099 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.09676871799999, 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:46.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:46.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:46.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.105 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:46.107 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:46.107 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 204.096118037, 2023-12-06T08:00:46.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:46.107 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.108 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:46.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:46.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:46.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:46.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:46.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:46.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:46.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.09264534600001, 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:46.119 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:46.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:46.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.091506624, 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.09055892699999, 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:46.130 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:46.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:46.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:46.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:46.135 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.062194933, 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:46.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.03398541600001, 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:46.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:46.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:46.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.42378748199999, 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:46.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:46.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.42314264199999, 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.402675888, 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.156 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:46.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.40192697200001, 2023-12-06T08:00:46.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.166 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.401284903, 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:46.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:46.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:46.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:46.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.172 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.172 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.172 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:46.172 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:46.172 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:46.172 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:46.028+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:46.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:46.175 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:46.174+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:46.275 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:46.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:46.347 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:46.347 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:46.347 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:46.347 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:46.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:46.401 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:46.401+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:46.477 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:46.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:46.486 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:46.486 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:46.486 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:46.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:46.581 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:46.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:46.665 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:46.665 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:46.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:46.666 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:46.666 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:46.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:46.716 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:46.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:46.780 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:46.780+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:46.846 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:46.846 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:46.846 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:46.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:46.882 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:46.880+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:46.982 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:46.981+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:47.025 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:47.025 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:47.025 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:47.026 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:47.026 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:47.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:47.083 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:47.083+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:47.171 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:47.171 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:47.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:47.171 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.75141446500001, 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:47.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:47.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:47.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:47.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.5879156999999999, 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:47.179 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:47.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:00:47.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:00:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:47.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:47.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:00:47.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:47.186 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:47.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:47.190 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:47.188+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:47.289 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:47.289+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:47.319 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:47.319 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:47.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:47.413 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:47.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:47.492 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:47.491+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:47.500 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:47.501 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:47.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:47.598 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:47.598+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:47.683 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:47.684 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:47.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:47.699 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:47.698+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:47.799 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:47.798+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:47.841 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:47.841 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:47.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:47.899 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:47.899+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:48.000 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:48.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:48.037 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:48.038 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:48.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:48.102 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:48.102+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:48.203 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:48.203+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 207.239435402, 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.57565440100001, 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.57237377199999, 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.26759715599999, 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.26675405899999, 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.244218553, 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.24222613399999, 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.24169386899999, 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.22476896000001, 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.22393232600001, 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.19948505799999, 2023-12-06T08:00:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.19358269400001, 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.188876698, 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.186574823, 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.18607650000001, 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.18562258899999, 2023-12-06T08:00:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.184400189, 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 199.53923420800001, 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 199.53179579299999, 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 199.52172242099999, 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:48.317 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:48.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:48.319 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:48.318+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:48.420 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:48.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:48.488 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:48.488 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:48.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:48.521 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:48.521+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:48.622 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:48.622+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:48.670 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:48.670 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:48.670 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:48.670 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:48.670 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:48.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:48.723 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:48.722+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:48.825 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:48.824+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:48.849 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:48.849 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:48.849 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:48.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:48.927 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:48.926+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:49.027 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:49.027 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:49.027 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:49.027 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:49.028 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:49.027+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:49.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:49.131 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:49.131+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:49.208 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:49.208 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:49.208 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:49.208 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:49.208 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:49.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:49.232 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:49.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:49.334 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:49.333+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 188.772184791, 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:49.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:49.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6550074450000007, 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:49.354 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:49.357 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:49.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:49.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:49.358 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:49.358 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:49.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:49.435 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:49.435+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:49.489 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:49.489 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:49.489 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:49.490 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:49.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:49.536 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:49.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:49.626 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:49.626 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:49.626 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:49.626 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:49.626 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:49.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:49.638 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:49.637+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:49.738 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:49.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:49.767 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:49.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:49.860 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:49.860+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:49.940 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:49.940+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:49.960 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:49.961 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:49.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:50.055 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:50.054+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:50.142 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:50.142 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:50.142 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:50.142 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:50.143 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:50.143 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:50.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:50.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:50.243 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:50.243+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.261063361, 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.78994990800001, 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.559757529, 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.55904998, 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.55841298600001, 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.501347791, 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.49985702000001, 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.46750735800001, 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.46087658299999, 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.46022590199999, 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.449 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:50.392+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.45675321100001, 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.455614489, 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:50.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.45466679200001, 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.467 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:50.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:50.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:50.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:50.473 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.42630279799999, 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:50.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:50.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.398093281, 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.479 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.78789534699999, 2023-12-06T08:00:50.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.78725050700001, 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.491 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:50.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:50.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.766783753, 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.497 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.76603483700001, 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:50.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.765392768, 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:50.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:50.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:50.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:50.514 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:50.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:50.516 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:50.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:50.617 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:50.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:50.645 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:50.645 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:50.645 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:50.645 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:50.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:50.718 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:50.718+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:50.788 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:50.788 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:50.788 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:50.788 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:50.789 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:50.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:50.819 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:50.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:50.920 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:50.919+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:50.974 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:50.974 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:50.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:50.974 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:50.974 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:50.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:51.021 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:51.021+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:51.121 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:51.121 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:51.121 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:51.121 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:51.122 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:51.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:51.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:51.223 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:51.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:51.259 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:51.259 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:51.259 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:51.259 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:51.259 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:51.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:51.327 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:51.324+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:51.401 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.982745134, 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:51.402 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:51.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:51.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:51.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.819246369, 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:51.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:51.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:51.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:51.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:51.413 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:51.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:51.456 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:51.456+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:51.526 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:51.525+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:51.546 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:51.546 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:51.546 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:51.546 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:51.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:51.626 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:51.626+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:51.695 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:51.695 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:51.695 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:51.695 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:51.696 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:51.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:51.751 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:51.750+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:51.828 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:51.828+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:51.837 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:51.837 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:51.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:51.838 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:51.838 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:51.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:51.931 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:51.930+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:52.023 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:52.023 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:52.024 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:52.024 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:52.031 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:52.031+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:52.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:52.055 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:07.971 100790== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:52.055 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:07.967 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:52.132 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:52.132+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:52.185 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:52.186 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:52.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:52.236 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:52.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:52.286 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.197 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:52.296 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:08.212 100790== Warning: unimplemented fcntl command: 1036 2023-12-06T08:00:52.336 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:52.335+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 211.42466040299999, 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.760879402, 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.75759877300001, 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.45282215700001, 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.45197906000001, 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.42944355399999, 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.42745113500001, 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.42691887000001, 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:52.442 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.409993961, 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:52.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:52.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.446 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 210.409157327, 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:52.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:52.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:52.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.38471005900001, 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.451 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:52.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:52.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:52.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.455 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.37880769500001, 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:52.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:52.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.459 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.37410169899999, 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.37179982399999, 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.371301501, 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.37084759000001, 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.36962518999999, 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 203.724459209, 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 203.71702079400001, 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:52.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:52.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 203.70694742200001, 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:52.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:52.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:52.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:52.491 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:52.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:52.493 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:52.492+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:52.594 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:52.594+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:52.673 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:52.674 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:52.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:52.729 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:52.728+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:52.796 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:52.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:52.817 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:52.817 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:52.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:52.911 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:52.910+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:52.998 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:52.998+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:53.013 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:53.013 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:53.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:53.013 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:53.013 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:53.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:53.109 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:53.108+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:53.291 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:53.291 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:53.291 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:53.291 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:53.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:53.292 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:53.291+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:53.391 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:53.391+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:53.463 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:53.463 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:53.463 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:53.463 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:53.464 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:53.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:53.492 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:53.492+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:53.593 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:53.593+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 193.02883921700001, 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 12.911661871, 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:00:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:00:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:53.615 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:53.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:53.694 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:53.695+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:53.748 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:53.748 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:53.748 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:53.749 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:53.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:53.795 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:53.796+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:53.887 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:53.888 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:53.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:53.897 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:53.896+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:54.001 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:53.998+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:54.069 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:54.069 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:54.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:54.069 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:54.069 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:54.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:54.119 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:54.119+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:54.200 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:54.200+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:54.247 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:54.248 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:54.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:54.301 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:54.301+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:54.404 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:54.404+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:54.405 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:54.405 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:54.405 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:54.405 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:54.405 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:54.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:54.505 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:54.505+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:54.606 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:54.606+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.656 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 214.525600336, 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:54.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:54.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:54.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.663 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 213.05448688300001, 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:54.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.82429450399999, 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:54.669 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.670 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:54.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.823586955, 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:54.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.678 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:54.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:54.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:54.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:54.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:54.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.687 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.687 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:54.688 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.82294996100001, 2023-12-06T08:00:54.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:54.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:54.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:54.695 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.765884766, 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:54.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:54.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:54.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.76439399500001, 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:54.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.708 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.732044333, 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.72541355800001, 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:54.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.720 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.72476287699999, 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:54.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:54.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:54.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.721290186, 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:54.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:54.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.720151464, 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.731 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.71920376700001, 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:54.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:54.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:54.741 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.69083977299999, 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:54.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:54.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.662630256, 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.747 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.05243232199999, 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:54.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:54.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:54.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:54.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:54.757 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.05178748200001, 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.031320728, 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:54.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:54.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:54.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.767 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.03057181200001, 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.029929743, 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:54.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:54.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:54.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:54.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:54.778 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:54.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:54.780 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:54.780+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:54.881 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:54.881+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:54.964 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:54.964 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:54.964 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:54.964 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:54.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:54.982 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:54.983+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:55.083 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:55.083+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:55.149 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:55.150 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:55.150 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:55.150 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:55.150 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:55.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:55.184 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:55.184+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:55.285 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:55.285+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:55.288 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:55.289 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:55.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:55.386 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:55.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:55.469 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:55.470 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:55.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:55.487 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:55.487+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:55.588 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:55.588+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:55.689 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:55.689+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:55.696 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:55.696 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:55.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:55.790 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:55.790+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:55.887 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:55.888 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 196.46873042300001, 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:55.889 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:55.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.890 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:55.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:55.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:55.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:55.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:55.895 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 16.305231658, 2023-12-06T08:00:55.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:00:55.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:55.897 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:55.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:55.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:55.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:55.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:00:55.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:00:55.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:55.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:00:55.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:00:55.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:00:55.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:00:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:55.912 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:55.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:55.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:55.912 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:55.912 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:55.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:55.915 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:55.915+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:56.015 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:56.015+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:56.046 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.046 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:56.046 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:56.046 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:00:56.116 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:56.116+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:56.184 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.184 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:56.184 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:56.184 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:56.184 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:00:56.217 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:56.217+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:56.319 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:56.319+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:56.321 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.321 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:56.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:56.322 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:56.322 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:00:56.420 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:56.420+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:56.466 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.466 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:56.466 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:56.466 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:00:56.522 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:56.522+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:56.603 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.603 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:56.604 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:56.604 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:56.604 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:00:56.623 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:56.623+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:56.724 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:56.724+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:56.823 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:56.823 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:56.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:56.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:56.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.84198277900001, 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.17820177799999, 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:00:56.828 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.174921149, 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:00:56.832 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:00:56.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:00:56.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:00:56.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:00:56.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:00:56.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.870144533, 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:00:56.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.845 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.869301436, 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:00:56.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:00:56.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.84676593, 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:00:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:00:56.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:00:56.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.844773511, 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.855 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:00:56.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:00:56.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.844241246, 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.859 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.82731633700001, 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.82647970299999, 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:00:56.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.871 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.802032435, 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:00:56.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.79613007099999, 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:00:56.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.79142407500001, 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.78912220000001, 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.78862387699999, 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.788169966, 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:00:56.896 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.78694756600001, 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:00:56.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:00:56.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.900 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 208.14178158499999, 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:00:56.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 208.13434316999999, 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:00:56.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 208.124269798, 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:56.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:00:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.912 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.913 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:56.913 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:56.913 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:56.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:00:56.915 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:56.915+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:57.015 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:57.016+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:57.084 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.084 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:57.084 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:57.084 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:00:57.135 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:57.135+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:57.218 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:00:57.218+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:00:57.261 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:57.261 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:00:57.318 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:57.318+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:57.419 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:57.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:57.441 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:00:57.442 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:00:57.520 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:57.521+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:57.578 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.578 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:57.578 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:00:57.578 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:00:57.622 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:57.621+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:57.722 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:57.722+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:57.756 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.756 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:00:57.756 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:00:57.756 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:00:57.756 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:00:57.823 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:57.823+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 197.31908798200001, 2023-12-06T08:00:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 17.201910636000001, 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:00:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:00:57.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:00:57.904 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:00:57.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:00:57.926 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:57.925+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:58.026 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:58.026+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:58.045 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:58.045 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:00:58.128 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:58.128+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:58.181 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.182 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:58.182 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:58.182 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:58.182 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:00:58.234 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:58.234+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:58.318 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.318 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:58.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:58.318 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:58.318 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:00:58.368 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:58.368+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:58.430 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:58.430+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:58.498 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:58.498 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:00:58.548 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:00:58.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:00:58.630 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.631 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:58.631 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:58.631 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:58.631 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.631 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:58.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:58.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:00:58.732 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:58.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:58.833 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:58.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:00:58.861 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.732169336, 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.862 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 217.26105588300001, 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.870 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.870 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:00:58.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.870 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:00:58.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 217.030863504, 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 217.030155955, 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.882 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:00:58.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 217.02951896100001, 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:00:58.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:00:58.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.972453766, 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:00:58.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:00:58.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.97096299500001, 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.900 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:58.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:00:58.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:00:58.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:00:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.93861333300001, 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:00:58.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:00:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:00:58.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.93198255799999, 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.911 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.912 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:00:58.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.912 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.93133187699999, 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.917 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:00:58.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:00:58.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 215.92785918600001, 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:00:58.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:00:58.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:00:58.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.927 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 215.926720464, 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:00:58.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 215.92577276700001, 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.933 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:00:58.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:00:58.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 215.897408773, 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.938 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 215.869199256, 2023-12-06T08:00:58.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:00:58.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:00:58.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:00:58.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:58.948 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.25900132199999, 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:00:58.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.25835648200001, 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.954 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:00:58.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:00:58.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.237889728, 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.959 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.23714081200001, 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:58.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:00:58.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.236498743, 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:58.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:00:58.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:58.975 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:58.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:00:58.977 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:58.977+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:59.078 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:00:59.077+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:00:59.145 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:59.145 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:00:59.179 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:00:59.179+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:00:59.280 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:59.280+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:59.324 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.324 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:59.324 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:59.324 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:59.324 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:00:59.381 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:59.381+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:59.463 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:59.464 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:00:59.464 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:00:59.481 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:59.482+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:59.583 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:59.583+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:59.644 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:00:59.644 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:00:59.684 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:00:59.684+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:00:59.784 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.784 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:00:59.784 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:00:59.784 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:00:59.784 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.784 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:00:59.785+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:00:59.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:00:59.885 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:00:59.885+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:00:59.930 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 200.51210144300001, 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:59.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:59.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 20.348602677999999, 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:00:59.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:00:59.938 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:00:59.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:00:59.985 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:00:59.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:00.080 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.080 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:00.080 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:00.080 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.087 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:00.087+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:00.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:00.187 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:00.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:00.217 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.217 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:00.217 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:00.218 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:00.218 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:00.289 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:00.289+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:00.353 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.353 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:00.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:00.353 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:00.353 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:00.403 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:00.404+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:00.490 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:00.490+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:00.530 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.530 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:00.530 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:00.530 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:00.591 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:00.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:00.665 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.665 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:00.665 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:00.665 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:00.665 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:00.692 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:00.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:00.792 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:00.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.90227090100001, 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:00.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:00.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:00.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.23848989999999, 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.235209271, 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.930432655, 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:00.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.929589558, 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:00.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:00.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.90705405200001, 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:00.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:00.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.907 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.905061633, 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:00.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:00.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.911 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.904529368, 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:00.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:00.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:00.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.88760445899999, 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.916 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:00.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:00.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:00.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:00.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.88676782499999, 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.862320557, 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.856418193, 2023-12-06T08:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.85171219700001, 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.84941032200001, 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.84891199899999, 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.848458088, 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.84723568800001, 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 212.20206970699999, 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 212.194631292, 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 212.18455792, 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:00.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:00.969 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:00.968+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:01.069 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:01.069+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:01.138 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:01.138 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:01.170 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:01.170+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:01.271 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:01.271+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:01.298 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.298 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:01.298 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:01.298 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:01.298 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:01.372 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:01.372+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:01.438 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.438 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:01.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:01.438 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:01.438 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:01.473 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:01.473+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:01.574 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:01.574+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:01.577 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:01.578 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:01.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:01.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:01.675 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:01.675+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:01.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:01.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:01.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.713 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:01.713 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:01.713 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:01.713 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:01.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:01.794 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:01.807 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:01.808+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:01.879 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:01.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:01.895 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.316147941, 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:01.896 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:01.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:01.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.198970594999999, 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:01.909 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:01.910 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:01.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:01.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:01.979 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:01.980+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:01.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:01.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:02.063 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.064 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.066 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:02.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:02.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:02.079 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:02.080+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:02.153 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:02.154 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:02.180 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:02.180+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:02.207 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.207 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:02.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:02.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:02.298 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:02.298+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:02.373 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.374 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:02.385 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:02.383+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:02.485 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:02.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:02.561 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.561 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:02.586 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:02.586+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:02.682 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 0.189106 (3069) 2023-12-06T08:01:02.692 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:02.691+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:02.704 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.704 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:02.792 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:02.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:02.892 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:01:02.904 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:02.904+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:02.909 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:02.909 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:02.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:02.995 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:02.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:03.095 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.095 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.096 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:03.096+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:03.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:03.197 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:03.197+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:03.276 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.276 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:03.299 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:03.299+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:03.400 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:03.400+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:03.414 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.415 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:03.447 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:03.461 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T08:01:03.461 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-06T08:01:03.462 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-06T08:01:03.501 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:03.501+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:03.505 INFO:tasks.ceph.mon.a:Started 2023-12-06T08:01:03.505 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T08:01:03.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:03.581 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.581 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:03.602 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:03.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:03.703 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:03.703+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:03.728 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.728 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:03.804 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:03.804+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:03.895 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:03.895 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:03.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:03.906 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:03.905+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:04.024 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:04.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:04.040 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.040 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:04.108 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:04.108+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:04.201 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:04.208 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:04.209+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:04.219 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.220 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:04.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:04.312 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:04.313+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:04.405 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.406 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.411 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:04.410+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:04.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:04.511 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:04.511+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:04.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:04.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:04.586 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.587 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:04.635 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:04.636 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:04.640 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:04.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:04.713 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:04.713+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:04.728 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.728 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:04.762 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:04.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:04.814 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:04.814+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:04.887 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:04.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:04.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:01:04.919 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:04.919 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:04.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:04.951 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:04.951+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:04.999 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:05.000 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:05.016 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:05.016+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:05.068 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:05.069 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:05.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:05.104 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:05.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:05.122 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:05.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:05.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:05.205 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:05.221 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:05.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:05.259 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:05.260 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:05.260 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:05.260 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:05.260 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:05.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:05.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:05.296 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:05.354 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:05.354+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:05.421 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:05.421+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:05.444 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:05.444 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:05.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:05.444 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:05.444 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:05.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:05.539 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:05.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:05.621 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:05.621 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:05.621 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:05.621 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:05.622 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:05.623+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:05.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:05.724 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:05.724+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:05.800 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:05.800 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:05.800 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:05.800 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:05.800 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:05.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:05.825 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:05.825+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:05.926 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:05.926+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:05.940 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:05.940 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:05.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:05.940 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 205.36185954300001, 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:05.941 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.942 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:05.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:05.946 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.244682196999999, 2023-12-06T08:01:05.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:05.948 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:05.948 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:05.948 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:05.948 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:05.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:05.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:05.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:05.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:05.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:05.957 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:05.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:06.027 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:06.026+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:06.090 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.090 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:06.090 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:06.090 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:06.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:06.142 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:06.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:06.228 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:06.228+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:06.272 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:06.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:06.329 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:06.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:06.419 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:06.419 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:06.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:06.470 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:06.470+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:06.522 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:01:06.532 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:06.532+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:06.610 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:06.611 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:06.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:06.633 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:06.633+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:06.734 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:06.734+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:06.751 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:06.752 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:06.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:06.835 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:06.834+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:06.970 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:06.969+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 226.839501842, 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:06.971 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:06.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:06.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.36838838899999, 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:06.977 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:06.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:06.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:06.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:06.984 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.13819601, 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:06.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.137488461, 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:06.994 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:06.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:06.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:06.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:06.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:06.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:07.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:07.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.13685146700001, 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.003 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:07.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:07.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.07978627200001, 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:07.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:07.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:07.018 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.07829550100001, 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:07.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:07.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:07.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.024 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.04594583900001, 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.025 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:07.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:07.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:07.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.03931506399999, 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:07.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:07.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.041 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.038664383, 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:07.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:07.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:07.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:07.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:07.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:07.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:07.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:07.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:07.055 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:07.055 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.055 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.03519169200001, 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:07.056 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:07.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:07.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:07.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:07.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:07.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:07.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:07.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:07.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:07.066 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:07.067 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.03405297, 2023-12-06T08:01:07.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:07.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:07.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:07.075 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.03310527299999, 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:07.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:07.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:07.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.004741279, 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:07.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:07.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:07.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:07.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:07.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:07.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:07.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:07.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:07.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:07.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:07.093 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 223.97653176200001, 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.094 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:07.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.36633382799999, 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.36568898799999, 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.345222234, 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.34447331800001, 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.343831249, 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:07.129 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:07.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:07.131 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:07.130+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:07.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:07.232 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:07.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:07.307 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:07.307 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:07.307 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:07.307 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:07.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:07.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:07.359 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:07.358+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:07.435 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:07.434+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:07.490 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:07.491 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:07.491 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:07.491 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:07.491 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:07.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:07.535 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:07.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:07.637 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:07.636+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:07.672 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:07.673 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:07.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:07.696 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:07.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:07.697 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T08:01:07.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T08:01:07.738 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:07.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:07.839 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:07.840+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:07.881 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:07.881 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:07.881 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:07.881 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:07.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:07.902 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:07.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:07.940 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:07.940+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:08.014 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:08.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:08.042 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:08.043+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:08.049 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:08.049 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:08.049 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:08.049 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:08.049 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:08.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:08.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:08.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:08.127 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:08.145 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:08.145+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:08.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:08.203 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:08.239 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.240 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.247 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:08.247+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:08.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:08.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:08.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:08.348 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:08.348+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:08.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:08.392 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:08.402 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.402 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:08.449 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:08.449+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:08.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:08.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 53, 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T08:01:06.336657+0000", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T08:01:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T08:01:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T08:01:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T08:01:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T08:01:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T08:01:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T08:01:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T08:01:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T08:01:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T08:01:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T08:01:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T08:01:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T08:01:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T08:01:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T08:01:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T08:01:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 3, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 3, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T08:01:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T08:01:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 30, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:08.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.517 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.520 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T08:01:08.521 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.522 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 51, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.523 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.524 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T08:01:08.525 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T08:01:08.526 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 45, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.528 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.530 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:08.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.532 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T08:01:08.533 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0.1891021728515625, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.534 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T08:01:08.538 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:08.539 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 24, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.540 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T08:01:08.541 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T08:01:08.542 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T08:01:08.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.544 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.545 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.546 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:08.548 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.0", 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: 4, 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.5", 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: 6 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.550 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 1 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.551 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T08:01:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T08:01:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T08:01:08.552 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:08.553 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:08.554 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:08.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:01:08.557 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.557 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.558 INFO:tasks.thrashosds.thrasher:j is {'epoch': 53, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T08:01:06.336657+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 0.1891021728515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [4, 3]}, {'pgid': '2.5', 'osds': [1, 6]}, {'pgid': '2.6', 'osds': [7, 2]}], 'pg_upmap_items': [{'pgid': '2.6', 'mappings': [{'from': 0, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T08:01:08.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:01:08.559 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:08.556+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:08.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:08.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:08.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:08.658 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:08.658+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:08.687 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.687 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:08.759 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:08.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:08.823 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.824 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:08.879 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:08.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:08.960 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:08.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:08.965 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:08.965 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:08.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:09.062 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:09.062+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:09.144 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.145 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:09.163 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:09.163+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:09.264 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:09.264+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:09.324 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.324 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:09.377 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:09.377+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:09.463 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.464 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.466 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:09.466+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:09.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:09.484 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:25.396 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.528 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:01:09.529 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:01:09.538 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:25.450 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.541 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2023-12-06T08:01:07.839174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221104,"kb_used_data":4924,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753616,"statfs":{"total":773094113280,"available":772867702784,"internally_reserved":0,"allocated":5042176,"data_stored":2363718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":94,"apply_latency_ms":94,"commit_latency_ns":94000000,"apply_latency_ns":94000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.886757+0000","last_change":"2023-12-06T07:57:30.896116+0000","last_active":"2023-12-06T08:01:02.886757+0000","last_peered":"2023-12-06T08:01:02.886757+0000","last_clean":"2023-12-06T08:01:02.886757+0000","last_became_active":"2023-12-06T07:57:30.890946+0000","last_became_peered":"2023-12-06T07:57:30.890946+0000","last_unstale":"2023-12-06T08:01:02.886757+0000","last_undegraded":"2023-12-06T08:01:02.886757+0000","last_fullsized":"2023-12-06T08:01:02.886757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:48:45.856269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.741589+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T08:01:02.741589+0000","last_peered":"2023-12-06T08:01:02.741589+0000","last_clean":"2023-12-06T08:01:02.741589+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T08:01:02.741589+0000","last_undegraded":"2023-12-06T08:01:02.741589+0000","last_fullsized":"2023-12-06T08:01:02.741589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.930339+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:01:02.930339+0000","last_peered":"2023-12-06T08:01:02.930339+0000","last_clean":"2023-12-06T08:01:02.930339+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:01:02.930339+0000","last_undegraded":"2023-12-06T08:01:02.930339+0000","last_fullsized":"2023-12-06T08:01:02.930339+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-06T08:01:04.371040+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T08:01:04.371040+0000","last_peered":"2023-12-06T08:01:04.371040+0000","last_clean":"2023-12-06T08:01:04.371040+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T08:01:04.371040+0000","last_undegraded":"2023-12-06T08:01:04.371040+0000","last_fullsized":"2023-12-06T08:01:04.371040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-06T08:01:04.450104+0000","last_change":"2023-12-06T08:01:04.450104+0000","last_active":"2023-12-06T08:01:04.450104+0000","last_peered":"2023-12-06T08:01:04.450104+0000","last_clean":"2023-12-06T08:01:04.450104+0000","last_became_active":"2023-12-06T08:01:04.441340+0000","last_became_peered":"2023-12-06T08:01:04.441340+0000","last_unstale":"2023-12-06T08:01:04.450104+0000","last_undegraded":"2023-12-06T08:01:04.450104+0000","last_fullsized":"2023-12-06T08:01:04.450104+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:26:50.232248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.739860+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T08:01:02.739860+0000","last_peered":"2023-12-06T08:01:02.739860+0000","last_clean":"2023-12-06T08:01:02.739860+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T08:01:02.739860+0000","last_undegraded":"2023-12-06T08:01:02.739860+0000","last_fullsized":"2023-12-06T08:01:02.739860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-06T08:01:03.886277+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:01:03.886277+0000","last_peered":"2023-12-06T08:01:03.886277+0000","last_clean":"2023-12-06T08:01:03.886277+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:01:03.886277+0000","last_undegraded":"2023-12-06T08:01:03.886277+0000","last_fullsized":"2023-12-06T08:01:03.886277+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":43,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.889935+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T08:01:02.889935+0000","last_peered":"2023-12-06T08:01:02.889935+0000","last_clean":"2023-12-06T08:01:02.889935+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T08:01:02.889935+0000","last_undegraded":"2023-12-06T08:01:02.889935+0000","last_fullsized":"2023-12-06T08:01:02.889935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":149,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-12-06T08:01:02.923364+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T08:01:02.923364+0000","last_peered":"2023-12-06T08:01:02.923364+0000","last_clean":"2023-12-06T08:01:02.923364+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T08:01:02.923364+0000","last_undegraded":"2023-12-06T08:01:02.923364+0000","last_fullsized":"2023-12-06T08:01:02.923364+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476784,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":195450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":9.4719999999999995,"15min":9.5510000000000002},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":19.495000000000001,"5min":27.638999999999999,"15min":27.638999999999999},"last":5.6369999999999996},{"interface":"front","average":{"1min":8.2789999999999999,"5min":8.7739999999999991,"15min":8.6639999999999997},"min":{"1min":5.5670000000000002,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":10.311,"5min":24.085000000000001,"15min":24.085000000000001},"last":6.71}]},{"osd":1,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":10.951000000000001,"5min":12.1,"15min":12.778},"min":{"1min":6.2089999999999996,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":20.888000000000002,"5min":29.213000000000001,"15min":29.213000000000001},"last":13.263999999999999},{"interface":"front","average":{"1min":11.334,"5min":13.771000000000001,"15min":14.41},"min":{"1min":6.6230000000000002,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":20.876000000000001,"5min":39.337000000000003,"15min":39.337000000000003},"last":10.701000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":12.481999999999999,"5min":12.294,"15min":12.522},"min":{"1min":7.7649999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":25.18,"5min":27.861000000000001,"15min":27.861000000000001},"last":16.356999999999999},{"interface":"front","average":{"1min":12.287000000000001,"5min":13.791,"15min":14.077},"min":{"1min":8.0510000000000002,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":17.773,"5min":26.388000000000002,"15min":26.388000000000002},"last":17.088999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":15.180999999999999,"5min":13.992000000000001,"15min":13.441000000000001},"min":{"1min":7.8609999999999998,"5min":7.8609999999999998,"15min":7.8609999999999998},"max":{"1min":27.263000000000002,"5min":27.263000000000002,"15min":27.263000000000002},"last":10.048999999999999},{"interface":"front","average":{"1min":16.231999999999999,"5min":16.158000000000001,"15min":16.135000000000002},"min":{"1min":9.1059999999999999,"5min":7.806,"15min":7.806},"max":{"1min":27.959,"5min":54.225999999999999,"15min":54.225999999999999},"last":13.987}]},{"osd":5,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":16.385000000000002,"5min":16.942,"15min":17.129999999999999},"min":{"1min":9.9819999999999993,"5min":8.8580000000000005,"15min":8.8580000000000005},"max":{"1min":50.066000000000003,"5min":50.066000000000003,"15min":50.066000000000003},"last":20.085999999999999},{"interface":"front","average":{"1min":18.908000000000001,"5min":17.34,"15min":17.378},"min":{"1min":11.112,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":50.478000000000002,"5min":50.478000000000002,"15min":50.478000000000002},"last":18.936}]},{"osd":6,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":13.637,"5min":14.238,"15min":14.647},"min":{"1min":8.4640000000000004,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":20.715,"5min":27.890000000000001,"15min":27.890000000000001},"last":21.663},{"interface":"front","average":{"1min":17.658000000000001,"5min":18.696000000000002,"15min":18.776},"min":{"1min":13.282,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":23.527000000000001,"5min":44.255000000000003,"15min":44.255000000000003},"last":20.957999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:00:27 2023","interfaces":[{"interface":"back","average":{"1min":15.627000000000001,"5min":18.381,"15min":19.100999999999999},"min":{"1min":8.0329999999999995,"5min":8.0329999999999995,"15min":8.0329999999999995},"max":{"1min":23.922000000000001,"5min":40.832000000000001,"15min":40.832000000000001},"last":12.436999999999999},{"interface":"front","average":{"1min":16.407,"5min":17.991,"15min":18.495000000000001},"min":{"1min":11.115,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":26.780999999999999,"5min":39.331000000000003,"15min":39.331000000000003},"last":17.579000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476785,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":195450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":8.9789999999999992,"5min":9.8879999999999999,"15min":10.294},"min":{"1min":4.8120000000000003,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":13.537000000000001,"5min":22.759,"15min":22.759},"last":9.9550000000000001},{"interface":"front","average":{"1min":8.452,"5min":10.244,"15min":10.488},"min":{"1min":5.3010000000000002,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":14.109999999999999,"5min":34.981000000000002,"15min":34.981000000000002},"last":8.2360000000000007}]},{"osd":1,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":12.387,"5min":18.638000000000002,"15min":21.207999999999998},"min":{"1min":9.2910000000000004,"5min":6.7800000000000002,"15min":6.7800000000000002},"max":{"1min":17.524000000000001,"5min":165.68600000000001,"15min":165.68600000000001},"last":10.565},{"interface":"front","average":{"1min":12.255000000000001,"5min":18.710000000000001,"15min":21.757000000000001},"min":{"1min":6.1109999999999998,"5min":6.1109999999999998,"15min":6.1109999999999998},"max":{"1min":19.568000000000001,"5min":153.68199999999999,"15min":153.68199999999999},"last":11.233000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":10.132,"5min":10.574999999999999,"15min":11.039},"min":{"1min":6.0720000000000001,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":25.210000000000001,"5min":35.935000000000002,"15min":35.935000000000002},"last":26.887},{"interface":"front","average":{"1min":14.794,"5min":15.691000000000001,"15min":15.704000000000001},"min":{"1min":8.7189999999999994,"5min":8.6560000000000006,"15min":8.6560000000000006},"max":{"1min":26.721,"5min":30.433,"15min":30.433},"last":25.497}]},{"osd":4,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":11.673,"5min":14.916,"15min":16.321999999999999},"min":{"1min":7.1829999999999998,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":17.850000000000001,"5min":50.137999999999998,"15min":50.137999999999998},"last":28.527999999999999},{"interface":"front","average":{"1min":15.606999999999999,"5min":16.504999999999999,"15min":16.574999999999999},"min":{"1min":9.5510000000000002,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":24.564,"5min":46.325000000000003,"15min":46.325000000000003},"last":41.265000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":17.285,"5min":19.646000000000001,"15min":20.302},"min":{"1min":10.565,"5min":10.106,"15min":10.106},"max":{"1min":23.98,"5min":50.320999999999998,"15min":50.320999999999998},"last":42.633000000000003},{"interface":"front","average":{"1min":13.266,"5min":16.234999999999999,"15min":17.268000000000001},"min":{"1min":7.9820000000000002,"5min":7.9370000000000003,"15min":7.9370000000000003},"max":{"1min":20.184000000000001,"5min":44.323,"15min":44.323},"last":25.114000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":18.402999999999999,"15min":19.859000000000002},"min":{"1min":8.6579999999999995,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":31.263000000000002,"5min":104.834,"15min":104.834},"last":44.640999999999998},{"interface":"front","average":{"1min":18.785,"5min":20.640999999999998,"15min":20.742000000000001},"min":{"1min":11.308999999999999,"5min":11.308999999999999,"15min":11.308999999999999},"max":{"1min":30.382000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":43.287999999999997}]},{"osd":7,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":15.943,"5min":18.603000000000002,"15min":19.495999999999999},"min":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":25.931999999999999,"5min":42.915999999999997,"15min":42.915999999999997},"last":26.190999999999999},{"interface":"front","average":{"1min":16.446999999999999,"5min":18.324999999999999,"15min":19.062999999999999},"min":{"1min":10.57,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":31.873000000000001,"5min":42.506,"15min":42.506},"last":43.697000000000003}]}]},{"osd":1,"up_from":16,"seq":68719476785,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":929792,"data_stored":593290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":8.6349999999999998,"5min":11.878,"15min":13.079000000000001},"min":{"1min":4.835,"5min":4.835,"15min":4.835},"max":{"1min":23.859999999999999,"5min":80.640000000000001,"15min":80.640000000000001},"last":13.811},{"interface":"front","average":{"1min":10.516,"5min":10.895,"15min":11.406000000000001},"min":{"1min":4.7999999999999998,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":24.472000000000001,"5min":25.390000000000001,"15min":25.390000000000001},"last":16.184999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":13.01,"5min":18.024999999999999,"15min":20.152999999999999},"min":{"1min":7.0940000000000003,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":32.081000000000003,"5min":100.703,"15min":100.703},"last":14.615},{"interface":"front","average":{"1min":11.221,"5min":16.643000000000001,"15min":18.620000000000001},"min":{"1min":4.867,"5min":4.867,"15min":4.867},"max":{"1min":28.448,"5min":95.960999999999999,"15min":95.960999999999999},"last":6.0289999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":15.468,"5min":17.209,"15min":18.189},"min":{"1min":10.103999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":22.573,"5min":42.167999999999999,"15min":42.167999999999999},"last":15.398999999999999},{"interface":"front","average":{"1min":9.7230000000000008,"5min":12.231,"15min":13.391},"min":{"1min":6.2460000000000004,"5min":6.2460000000000004,"15min":6.2460000000000004},"max":{"1min":15.359999999999999,"5min":46.613,"15min":46.613},"last":7.7889999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":11.882999999999999,"5min":16.734000000000002,"15min":18.109999999999999},"min":{"1min":7.2720000000000002,"5min":7.2720000000000002,"15min":7.2720000000000002},"max":{"1min":23.521999999999998,"5min":81.838999999999999,"15min":81.838999999999999},"last":8.7240000000000002},{"interface":"front","average":{"1min":16.670000000000002,"5min":19.699000000000002,"15min":21.103000000000002},"min":{"1min":10.754,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":24.308,"5min":66.515000000000001,"15min":66.515000000000001},"last":16.966999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":14.048,"5min":15.321,"15min":16.195},"min":{"1min":8.7929999999999993,"5min":6.351,"15min":6.351},"max":{"1min":27.821000000000002,"5min":56.235999999999997,"15min":56.235999999999997},"last":10.052},{"interface":"front","average":{"1min":15.279999999999999,"5min":18.651,"15min":19.547000000000001},"min":{"1min":10.997999999999999,"5min":9.766,"15min":9.766},"max":{"1min":26.425000000000001,"5min":82.381,"15min":82.381},"last":7.1680000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":14.981999999999999,"5min":17.045000000000002,"15min":18.300999999999998},"min":{"1min":8.8919999999999995,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":29.782,"5min":58.619,"15min":58.619},"last":12.782},{"interface":"front","average":{"1min":16.030000000000001,"5min":19.873000000000001,"15min":20.975000000000001},"min":{"1min":7.8529999999999998,"5min":7.8529999999999998,"15min":7.8529999999999998},"max":{"1min":25.824000000000002,"5min":83.632000000000005,"15min":83.632000000000005},"last":9.343}]},{"osd":7,"last update":"Wed Dec 6 08:00:28 2023","interfaces":[{"interface":"back","average":{"1min":19.091999999999999,"5min":21.437000000000001,"15min":22.501999999999999},"min":{"1min":13.81,"5min":13.016,"15min":13.016},"max":{"1min":26.702000000000002,"5min":62.006999999999998,"15min":62.006999999999998},"last":17.806000000000001},{"interface":"front","average":{"1min":17.151,"5min":19.126000000000001,"15min":20.373999999999999},"min":{"1min":10.784000000000001,"5min":9.9740000000000002,"15min":9.9740000000000002},"max":{"1min":30.812000000000001,"5min":60.698999999999998,"15min":60.698999999999998},"last":10.711}]}]},{"osd":0,"up_from":16,"seq":68719476786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":195450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":10.210000000000001,"5min":16.306999999999999,"15min":19.251999999999999},"min":{"1min":6.1360000000000001,"5min":5.6470000000000002,"15min":5.6470000000000002},"max":{"1min":19.058,"5min":252.88399999999999,"15min":252.88399999999999},"last":8.7070000000000007},{"interface":"front","average":{"1min":9.827,"5min":16.876999999999999,"15min":20.216999999999999},"min":{"1min":5.867,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":19.094000000000001,"5min":274.91500000000002,"15min":274.91500000000002},"last":7.7569999999999997}]},{"osd":2,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":11.872999999999999,"5min":15.353999999999999,"15min":16.957999999999998},"min":{"1min":7.194,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":23.404,"5min":82.866,"15min":82.866},"last":10.938000000000001},{"interface":"front","average":{"1min":15.375,"5min":15.722,"15min":16.294},"min":{"1min":8.125,"5min":6.7930000000000001,"15min":6.7930000000000001},"max":{"1min":31.998999999999999,"5min":85.991,"15min":85.991},"last":9.798}]},{"osd":3,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":10.512,"5min":13.564,"15min":14.827999999999999},"min":{"1min":6.6859999999999999,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":20.039000000000001,"5min":76.593000000000004,"15min":76.593000000000004},"last":8.4290000000000003},{"interface":"front","average":{"1min":15.696999999999999,"5min":18.558,"15min":20.228000000000002},"min":{"1min":7.5890000000000004,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":35.015000000000001,"5min":85.162000000000006,"15min":85.162000000000006},"last":12.904}]},{"osd":4,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":18.378,"5min":17.077000000000002,"15min":17.411000000000001},"min":{"1min":7.8019999999999996,"5min":7.516,"15min":7.516},"max":{"1min":76.387,"5min":76.387,"15min":76.387},"last":13.529},{"interface":"front","average":{"1min":12.448,"5min":13.135,"15min":13.776999999999999},"min":{"1min":7.4089999999999998,"5min":6.5780000000000003,"15min":6.5780000000000003},"max":{"1min":24.988,"5min":70.085999999999999,"15min":70.085999999999999},"last":10.375}]},{"osd":5,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":16.922000000000001,"5min":24.100000000000001,"15min":26.948},"min":{"1min":8.2119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":31.056000000000001,"5min":144.60499999999999,"15min":144.60499999999999},"last":14.233000000000001},{"interface":"front","average":{"1min":18.577999999999999,"5min":23.274999999999999,"15min":25.561},"min":{"1min":9.4380000000000006,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.622999999999998,"5min":142.25,"15min":142.25},"last":15.621}]},{"osd":6,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":15.645,"5min":16.806999999999999,"15min":17.891999999999999},"min":{"1min":7.2729999999999997,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":26.760000000000002,"5min":75.471999999999994,"15min":75.471999999999994},"last":23.998999999999999},{"interface":"front","average":{"1min":17.183,"5min":21.494,"15min":23.484000000000002},"min":{"1min":11.494,"5min":9.4589999999999996,"15min":9.4589999999999996},"max":{"1min":27.792999999999999,"5min":84.617000000000004,"15min":84.617000000000004},"last":25.506}]},{"osd":7,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":20.849,"5min":21.021000000000001,"15min":21.827000000000002},"min":{"1min":10.140000000000001,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":78.506,"5min":84.031000000000006,"15min":84.031000000000006},"last":16.510999999999999},{"interface":"front","average":{"1min":15.989000000000001,"5min":17.646999999999998,"15min":18.757000000000001},"min":{"1min":10.925000000000001,"5min":9.6310000000000002,"15min":9.6310000000000002},"max":{"1min":33.088000000000001,"5min":77.974000000000004,"15min":77.974000000000004},"last":12.262}]}]},{"osd":4,"up_from":15,"seq":64424509490,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":540672,"data_stored":199888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":7.1219999999999999,"5min":8.6839999999999993,"15min":9.3729999999999993},"min":{"1min":4.3890000000000002,"5min":4.234,"15min":4.234},"max":{"1min":14.000999999999999,"5min":23.677,"15min":23.677},"last":10.004},{"interface":"front","average":{"1min":8.9339999999999993,"5min":9.9109999999999996,"15min":10.164999999999999},"min":{"1min":5.7889999999999997,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":14.289999999999999,"5min":22.443999999999999,"15min":22.443999999999999},"last":5.7270000000000003}]},{"osd":1,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":9.2349999999999994,"5min":11.028,"15min":11.696},"min":{"1min":6.7190000000000003,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":14.755000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":6.7599999999999998},{"interface":"front","average":{"1min":11.24,"5min":13.287000000000001,"15min":13.797000000000001},"min":{"1min":7.4530000000000003,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":15.805999999999999,"5min":31.027000000000001,"15min":31.027000000000001},"last":11.395}]},{"osd":2,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":13.586,"5min":12.609,"15min":12.214},"min":{"1min":9.0990000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"max":{"1min":27.353000000000002,"5min":27.353000000000002,"15min":27.353000000000002},"last":14.972},{"interface":"front","average":{"1min":14.518000000000001,"5min":14.791,"15min":14.680999999999999},"min":{"1min":10.34,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":26.34,"5min":32.915999999999997,"15min":32.915999999999997},"last":8.7430000000000003}]},{"osd":3,"last update":"Wed Dec 6 08:00:29 2023","interfaces":[{"interface":"back","average":{"1min":12.747,"5min":11.598000000000001,"15min":11.522},"min":{"1min":6.8869999999999996,"5min":6.843,"15min":6.843},"max":{"1min":28.427,"5min":28.427,"15min":28.427},"last":13.704000000000001},{"interface":"front","average":{"1min":14.363,"5min":14.464,"15min":14.198},"min":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":23.417000000000002,"5min":25.077999999999999,"15min":25.077999999999999},"last":12.726000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:00:23 2023","interfaces":[{"interface":"back","average":{"1min":18.542000000000002,"5min":17.234000000000002,"15min":17.143999999999998},"min":{"1min":10.930999999999999,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":53.899999999999999,"5min":53.899999999999999,"15min":53.899999999999999},"last":10.628},{"interface":"front","average":{"1min":18.829000000000001,"5min":18.02,"15min":17.786000000000001},"min":{"1min":10.933,"5min":9.9399999999999995,"15min":9.9399999999999995},"max":{"1min":55.914000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":16.404}]},{"osd":6,"last update":"Wed Dec 6 08:00:23 2023","interfaces":[{"interface":"back","average":{"1min":15.651,"5min":20.079999999999998,"15min":21.661999999999999},"min":{"1min":11.720000000000001,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":22.193000000000001,"5min":93.073999999999998,"15min":93.073999999999998},"last":15.669},{"interface":"front","average":{"1min":12.247,"5min":16.23,"15min":17.885000000000002},"min":{"1min":8.6189999999999998,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":19.315999999999999,"5min":91.075999999999993,"15min":91.075999999999993},"last":18.471}]},{"osd":7,"last update":"Wed Dec 6 08:00:23 2023","interfaces":[{"interface":"back","average":{"1min":14.414,"5min":15.789,"15min":16.175000000000001},"min":{"1min":10.874000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":18.542000000000002,"5min":33.988999999999997,"15min":33.988999999999997},"last":17.106999999999999},{"interface":"front","average":{"1min":16.536000000000001,"5min":17.266999999999999,"15min":16.908999999999999},"min":{"1min":10.885,"5min":10.885,"15min":10.885},"max":{"1min":19.300000000000001,"5min":32.014000000000003,"15min":32.014000000000003},"last":13.369}]}]},{"osd":6,"up_from":14,"seq":60129542192,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":195450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":8.2530000000000001,"5min":9.6120000000000001,"15min":10.077999999999999},"min":{"1min":4.6470000000000002,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":17.477,"5min":70.965000000000003,"15min":70.965000000000003},"last":6.1970000000000001},{"interface":"front","average":{"1min":10.081,"5min":11.151999999999999,"15min":11.942},"min":{"1min":6.141,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":26.015999999999998,"5min":73.582999999999998,"15min":73.582999999999998},"last":7.0960000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":15.196,"5min":17.427,"15min":18.847999999999999},"min":{"1min":5.3929999999999998,"5min":4.383,"15min":4.383},"max":{"1min":70.049000000000007,"5min":217.595,"15min":217.595},"last":9.8480000000000008},{"interface":"front","average":{"1min":12.337,"5min":18.952999999999999,"15min":21.843},"min":{"1min":7.7709999999999999,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":27.167000000000002,"5min":284.96499999999997,"15min":284.96499999999997},"last":13.686999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":15.614000000000001,"5min":14.007,"15min":13.923},"min":{"1min":6.2069999999999999,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":42.404000000000003,"5min":46.024999999999999,"15min":46.024999999999999},"last":9.0370000000000008},{"interface":"front","average":{"1min":14.913,"5min":15.680999999999999,"15min":16.548999999999999},"min":{"1min":7.7610000000000001,"5min":7.7610000000000001,"15min":7.7610000000000001},"max":{"1min":44.386000000000003,"5min":46.966999999999999,"15min":46.966999999999999},"last":16.369}]},{"osd":3,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":19.173999999999999,"5min":16.268999999999998,"15min":15.987},"min":{"1min":10.567,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":65.719999999999999,"5min":65.719999999999999,"15min":65.719999999999999},"last":10.938000000000001},{"interface":"front","average":{"1min":17.507999999999999,"5min":20.861000000000001,"15min":23.190999999999999},"min":{"1min":8.1969999999999992,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":69.445999999999998,"5min":189.84999999999999,"15min":189.84999999999999},"last":17.289999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":23.059999999999999,"5min":17.199000000000002,"15min":16.321999999999999},"min":{"1min":9.3409999999999993,"5min":9.3409999999999993,"15min":9.3409999999999993},"max":{"1min":66.349999999999994,"5min":66.349999999999994,"15min":66.349999999999994},"last":19.530999999999999},{"interface":"front","average":{"1min":18.407,"5min":15.49,"15min":14.916},"min":{"1min":10.023999999999999,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":39.520000000000003,"5min":39.520000000000003,"15min":39.520000000000003},"last":11.772}]},{"osd":5,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":17.809000000000001,"5min":18.733000000000001,"15min":18.859999999999999},"min":{"1min":10.471,"5min":10.471,"15min":10.471},"max":{"1min":32.976999999999997,"5min":50.945,"15min":50.945},"last":18.117000000000001},{"interface":"front","average":{"1min":13.233000000000001,"5min":14.125999999999999,"15min":14.289999999999999},"min":{"1min":6.4960000000000004,"5min":6.4960000000000004,"15min":6.4960000000000004},"max":{"1min":29.141999999999999,"5min":51.948,"15min":51.948},"last":15.388}]},{"osd":7,"last update":"Wed Dec 6 08:00:21 2023","interfaces":[{"interface":"back","average":{"1min":17.920000000000002,"5min":16.32,"15min":15.946999999999999},"min":{"1min":9.1880000000000006,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":29.731999999999999,"5min":31.353999999999999,"15min":31.353999999999999},"last":12.470000000000001},{"interface":"front","average":{"1min":21.420000000000002,"5min":19.545999999999999,"15min":19.516999999999999},"min":{"1min":11.18,"5min":11.18,"15min":11.18},"max":{"1min":71.346999999999994,"5min":71.346999999999994,"15min":71.346999999999994},"last":18.925000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542193,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":929792,"data_stored":593290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":10.481999999999999,"5min":9.1920000000000002,"15min":9.1549999999999994},"min":{"1min":5.085,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":26.117999999999999,"5min":26.117999999999999,"15min":26.117999999999999},"last":9.0380000000000003},{"interface":"front","average":{"1min":10.065,"5min":9.5060000000000002,"15min":9.5909999999999993},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":6.1399999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":11.641,"5min":12.971,"15min":13.539999999999999},"min":{"1min":4.8659999999999997,"5min":4.8659999999999997,"15min":4.8659999999999997},"max":{"1min":21.096,"5min":67.347999999999999,"15min":67.347999999999999},"last":8.0920000000000005},{"interface":"front","average":{"1min":12.079000000000001,"5min":12.786,"15min":12.768000000000001},"min":{"1min":7.2670000000000003,"5min":5.8090000000000002,"15min":5.8090000000000002},"max":{"1min":19.670000000000002,"5min":39.052,"15min":39.052},"last":10.179}]},{"osd":2,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":10.069000000000001,"5min":12.188000000000001,"15min":12.144},"min":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":5.7629999999999999},"max":{"1min":15.087,"5min":31.873000000000001,"15min":31.873000000000001},"last":5.4649999999999999},{"interface":"front","average":{"1min":11.409000000000001,"5min":14.813000000000001,"15min":15.638},"min":{"1min":7.609,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":18.672000000000001,"5min":34.585000000000001,"15min":34.585000000000001},"last":16.887}]},{"osd":3,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":13.539,"5min":15.087,"15min":15.385},"min":{"1min":7.1340000000000003,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":21.683,"5min":30.355,"15min":30.355},"last":11.555},{"interface":"front","average":{"1min":12.829000000000001,"5min":14.711,"15min":15.035},"min":{"1min":6.4550000000000001,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":27.004999999999999,"5min":31.792000000000002,"15min":31.792000000000002},"last":10.786}]},{"osd":4,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":13.574999999999999,"5min":23.109000000000002,"15min":27.331},"min":{"1min":7.2969999999999997,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":19.408000000000001,"5min":334.86500000000001,"15min":334.86500000000001},"last":17.727},{"interface":"front","average":{"1min":13.548,"5min":23.271000000000001,"15min":27.276},"min":{"1min":9.3550000000000004,"5min":7.431,"15min":7.431},"max":{"1min":19.776,"5min":331.57900000000001,"15min":331.57900000000001},"last":12.255000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":16.988,"5min":18.629999999999999,"15min":19.593},"min":{"1min":9.1310000000000002,"5min":9.1310000000000002,"15min":9.1310000000000002},"max":{"1min":33.228000000000002,"5min":46.034999999999997,"15min":46.034999999999997},"last":18.515000000000001},{"interface":"front","average":{"1min":15.971,"5min":17.777000000000001,"15min":18.515999999999998},"min":{"1min":10.726000000000001,"5min":9.6709999999999994,"15min":9.6709999999999994},"max":{"1min":32.503999999999998,"5min":35.680999999999997,"15min":35.680999999999997},"last":13.744}]},{"osd":7,"last update":"Wed Dec 6 08:00:22 2023","interfaces":[{"interface":"back","average":{"1min":14.417999999999999,"5min":24.597000000000001,"15min":29.082999999999998},"min":{"1min":10.035,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":22.727,"5min":332.322,"15min":332.322},"last":13.176},{"interface":"front","average":{"1min":16.654,"5min":27.41,"15min":32.170000000000002},"min":{"1min":10.467000000000001,"5min":10.467000000000001,"15min":10.467000000000001},"max":{"1min":25.809999999999999,"5min":325.93599999999998,"15min":325.93599999999998},"last":19.387}]}]},{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":195450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":8.0690000000000008,"5min":9.1969999999999992,"15min":9.4390000000000001},"min":{"1min":5.1849999999999996,"5min":5.0449999999999999,"15min":5.0449999999999999},"max":{"1min":14.347,"5min":20.558,"15min":20.558},"last":6.1890000000000001},{"interface":"front","average":{"1min":11.199999999999999,"5min":11.968,"15min":11.978},"min":{"1min":7.7569999999999997,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":16.981000000000002,"5min":20.893999999999998,"15min":20.893999999999998},"last":8.2739999999999991}]},{"osd":1,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":14.542,"5min":15.5,"15min":16.390999999999998},"min":{"1min":7.2560000000000002,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":38.838999999999999,"5min":51.890000000000001,"15min":51.890000000000001},"last":12.195},{"interface":"front","average":{"1min":10.754,"5min":8.9030000000000005,"15min":8.5619999999999994},"min":{"1min":4.6529999999999996,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":37.042999999999999,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.0019999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":11.571999999999999,"5min":12.483000000000001,"15min":12.566000000000001},"min":{"1min":5.8220000000000001,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":30.379999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":20.170999999999999},{"interface":"front","average":{"1min":12.209,"5min":11.710000000000001,"15min":11.34},"min":{"1min":6.9889999999999999,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":35.002000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":18.934999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":12.000999999999999,"5min":23.582000000000001,"15min":27.969999999999999},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":27.995999999999999,"5min":228.17099999999999,"15min":228.17099999999999},"last":9.2769999999999992},{"interface":"front","average":{"1min":14.558,"5min":25.876000000000001,"15min":30.733000000000001},"min":{"1min":11.141,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":18.917999999999999,"5min":270.13,"15min":270.13},"last":14.448}]},{"osd":4,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":15.865,"15min":16.891999999999999},"min":{"1min":6.8220000000000001,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":23.157,"5min":86.980999999999995,"15min":86.980999999999995},"last":16.914999999999999},{"interface":"front","average":{"1min":14.502000000000001,"5min":18.971,"15min":20.207000000000001},"min":{"1min":7.8719999999999999,"5min":7.8719999999999999,"15min":7.8719999999999999},"max":{"1min":26.843,"5min":99.021000000000001,"15min":99.021000000000001},"last":10.026999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":17.593,"5min":20.943999999999999,"15min":21.789999999999999},"min":{"1min":10.254,"5min":10.254,"15min":10.254},"max":{"1min":33.235999999999997,"5min":97.191000000000003,"15min":97.191000000000003},"last":11.198},{"interface":"front","average":{"1min":16.475000000000001,"5min":19.873999999999999,"15min":20.951000000000001},"min":{"1min":9.3859999999999992,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":25.257000000000001,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.045999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:00:25 2023","interfaces":[{"interface":"back","average":{"1min":16.248000000000001,"5min":16.745999999999999,"15min":17.324999999999999},"min":{"1min":9.0950000000000006,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":32.805,"5min":74.064999999999998,"15min":74.064999999999998},"last":17.981000000000002},{"interface":"front","average":{"1min":19.295999999999999,"5min":21.353999999999999,"15min":22.204999999999998},"min":{"1min":10.055,"5min":10.055,"15min":10.055},"max":{"1min":34.284999999999997,"5min":92.477999999999994,"15min":92.477999999999994},"last":10.581}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:01:09.544 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [1, 7] 2023-12-06T08:01:09.544 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '1', '7'] 2023-12-06T08:01:09.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 1 7 2023-12-06T08:01:09.583 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:09.583+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:09.590 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:25.507 100790== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.654 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:25.571 100790== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.656 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.656 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:09.668 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:09.668+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:09.751 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:25.663 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.769 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:09.769+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:09.815 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.815 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:09.862 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:25.776 100790== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:09.871 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:09.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:09.963 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:09.963 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:09.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:09.973 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:09.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:10.073 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:10.073+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:10.143 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:10.143 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:10.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:10.197 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:10.197+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:10.275 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:10.275+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:10.313 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:10.314 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:10.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:10.343 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:01:10.400 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:10.400+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:10.477 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:10.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:10.492 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:10.492 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:10.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:10.579 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:10.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:10.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:10.673 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:10.674 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:10.680 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:10.680+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:10.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:10.781 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:10.780+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:10.859 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:10.859 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:10.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:10.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:10.879 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:10.884 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:10.883+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:10.972 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:10.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:10.985 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:10.984+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:11.009 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:11.009 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:11.009 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:11.009 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:11.009 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:11.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:11.059 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:11.060 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:11.104 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:11.104+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:11.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:11.166 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:11.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:11.186 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:11.186+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.13185024800001, 2023-12-06T08:01:11.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:11.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:11.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.274 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.66073679499999, 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.275 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:11.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:11.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:11.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.430544416, 2023-12-06T08:01:11.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:11.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:11.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:11.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.42983686700001, 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.288 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:11.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:11.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:11.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:11.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.42919987299999, 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.297 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.37213467800001, 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.37064390699999, 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.33829424499999, 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.33166347, 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.331012789, 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.32754009799999, 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.32640137600001, 2023-12-06T08:01:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.32545367899999, 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.297089685, 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.26888016800001, 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.658682234, 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.65803739399999, 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.63757064000001, 2023-12-06T08:01:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.63682172399999, 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.63617965500001, 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:11.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:11.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:11.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:11.407 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:11.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:11.408 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:11.405+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:11.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:11.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:11.507 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:11.507+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:11.557 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:11.557 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:11.595 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:11.595 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:11.595 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:11.595 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:11.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:11.630 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.7 pg_upmap mapping to [1,7] 2023-12-06T08:01:11.631 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:11.630+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:11.710 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:11.710+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:11.767 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:11.767 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:11.767 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:11.767 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:11.767 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:11.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:11.811 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:11.811+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:11.867 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:11.912 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:11.912+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:11.949 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:11.949 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:11.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:11.949 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:11.949 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:11.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:12.012 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:12.012+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:12.090 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:12.090 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:12.090 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:12.090 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:12.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:12.114 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:12.114+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:12.120 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.552 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:12.215 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:12.215+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:12.248 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:12.248 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:12.248 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:12.248 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:12.248 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:12.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:12.318 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:12.318+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:12.351 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:08.784 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:12.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:01:12.420 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:12.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 213.01517258000001, 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:12.437 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:12.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:12.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:12.440 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.851673814999998, 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.441 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:12.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:12.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:12.444 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:12.445 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:12.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:12.520 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:12.520+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:12.579 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:12.579 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:12.579 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:12.579 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:12.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:12.629 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:12.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:12.722 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:12.722+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:12.756 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:12.756 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:12.756 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:12.756 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:12.757 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:12.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:12.824 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:12.823+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:12.912 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:12.912 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:12.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:12.912 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:12.912 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:12.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:12.924 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:12.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:13.025 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:13.024+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:13.090 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.091 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:13.091 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:13.091 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:13.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:13.144 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:13.143+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:13.226 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.226 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:13.226 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:13.227 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:13.227 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:13.227 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:13.226+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:13.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:13.328 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:13.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:13.428 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:13.428+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 232.466718953, 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.449 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:13.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.80293795200001, 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:13.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.79965732299999, 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:13.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:13.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:13.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.460 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.49488070699999, 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.49403760999999, 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:13.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:13.468 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.471502104, 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.46950968499999, 2023-12-06T08:01:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:13.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:13.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:13.476 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.46897741999999, 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.45205251100001, 2023-12-06T08:01:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:13.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:13.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:13.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.45121587700001, 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:13.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:13.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.42676860899999, 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:13.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.42086624500001, 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:13.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:13.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.416160249, 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.413858374, 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:13.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.41336005100001, 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:13.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.41290613999999, 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.508 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.41168374, 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 224.76651775900001, 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:13.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:13.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 224.75907934400001, 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:13.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.523 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 224.74900597199999, 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:13.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:13.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:13.527 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:13.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:13.529 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:13.528+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:13.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:13.629 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:13.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:13.701 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.701 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:13.702 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:13.702 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:13.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:13.730 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:13.730+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:13.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:13.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:13.831 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:13.830+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:13.838 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.839 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:13.839 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:13.839 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:13.839 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:13.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:13.903 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:13.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:13.932 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:13.931+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:13.996 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:13.996 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:13.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:13.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:13.997 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:14.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:14.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:14.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:14.017 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:01:14.033 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:14.033+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:14.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:14.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:14.139 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:14.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:14.191 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:14.192 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:14.208 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:14.208 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:14.208 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:14.208 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:14.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:14.279 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:14.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:14.280 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:14.279+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:14.336 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:14.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:14.362 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:14.362 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:14.362 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:14.362 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:14.362 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:14.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:14.386 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:14.468 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:14.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:14.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:14.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:14.538 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:14.537+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:14.555 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:14.555 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:14.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:14.638 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:14.638+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:14.697 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:14.698 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:14.708 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:14.752 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:14.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:14.839 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:14.840 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:14.840 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:14.840+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:14.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:14.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:14.941 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:14.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:15.042 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:15.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:15.061 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:15.061 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:15.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:15.143 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:15.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:15.244 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:15.244+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:15.245 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:15.245 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:15.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:15.345 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:15.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:15.446 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:15.446+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:15.461 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:15.461 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:15.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:15.486 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:11.918 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:15.548 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:15.548+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:15.552 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:11.984 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:15.633 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:15.647 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:15.647 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:15.652 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:15.651+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:15.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:15.750 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:15.750+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:15.845 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:15.846 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:15.852 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:15.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:15.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:15.945 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:12.378 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:15.952 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:15.951+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:16.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:01:16.039 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.039 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:16.091 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:16.091+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:16.154 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:16.154+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:16.226 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.226 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:16.255 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:16.255+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:16.356 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:16.356+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:16.410 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.410 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:16.457 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:16.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:16.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:16.558 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:16.558+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:16.580 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.580 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:16.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:16.650 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:16.650 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-12-06T08:01:16.650 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T08:01:16.650 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T08:01:16.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-12-06T08:01:16.660 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:16.659+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:16.731 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.731 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:16.786 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:16.786 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T08:01:16.786 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:16.787 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:16.786+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:16.862 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:16.861+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:16.877 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:16.877 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:16.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:16.972 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:16.971+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:17.017 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:17.017 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:17.055 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:17.055 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:17.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:17.064 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:17.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:17.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:17.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:17.165 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:17.164+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:17.204 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:17.204 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:17.232 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:17.232 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:17.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:17.282 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:17.282+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:17.290 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:17.290 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:17.367 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:17.366+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:17.382 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:17.383 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:17.425 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:17.426 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:17.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:17.468 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:17.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:17.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:17.486 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:17.569 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:17.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:17.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:17.574 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:17.615 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:17.615 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:17.615 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:17.615 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:17.615 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:17.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:17.669 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:17.669+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:17.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:17.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:17.734 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:01:17.770 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:17.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:17.800 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 236.81761882500001, 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:17.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:17.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 236.15383782399999, 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:17.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 236.150557195, 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:17.810 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:17.814 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.84578057900001, 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:17.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.84493748200001, 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:17.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.82240197600001, 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.820409557, 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.819877292, 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:17.832 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.836 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.80295238299999, 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:17.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:17.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.802115749, 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:17.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:17.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:17.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.77766848100001, 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.852 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.771766117, 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:17.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:17.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:17.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.76706012099999, 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:17.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.861 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.76475824600001, 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:17.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:17.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.764259923, 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.763806012, 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:17.872 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:17.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 234.76258361199999, 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:17.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:17.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:17.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 229.117417631, 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:17.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:17.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:17.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 229.109979216, 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:17.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 229.09990584400001, 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:17.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:17.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:17.894 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:17.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:17.896 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:17.895+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:17.997 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:17.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:18.024 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.024 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:18.024 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:18.024 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:18.098 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:18.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:18.198 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:18.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:18.201 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:18.201 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:18.299 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:18.298+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:18.338 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:18.338 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:18.433 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:18.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:18.501 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:18.501+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:18.517 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.518 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:18.518 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:18.518 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:18.610 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:18.610+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:18.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:18.695 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.695 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:18.695 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:18.696 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:18.696 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.702 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:18.702+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:18.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:18.803 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:18.803+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:18.883 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.30460102500001, 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.187423678999998, 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:18.887 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:18.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:18.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:18.890 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:18.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:18.940 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:18.939+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:19.006 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:19.005+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:19.026 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:19.027 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:19.027 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:19.027 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:19.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:19.120 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:19.119+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:19.209 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:19.207+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:19.229 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:19.229 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:19.229 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:19.229 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:19.229 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:19.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:19.309 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:19.309+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:19.386 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:19.399 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T08:01:19.399 INFO:tasks.mon_recovery:removing mon b 2023-12-06T08:01:19.399 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T08:01:19.399 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:01:19.413 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:19.413 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:19.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:19.413 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:19.413 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:19.414 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:01:19.403+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 100786) UID: 0 2023-12-06T08:01:19.414 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:01:19.404+0000 9f54640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-06T08:01:19.414 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:19.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:19.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:19.511 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:19.510+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:19.613 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:19.612+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:19.628 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:19.628 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:19.628 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:19.628 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:19.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:19.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:19.715 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:19.714+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:19.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:01:19.815 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:19.815+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:19.825 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:19.825 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:19.825 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:19.825 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:19.825 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:19.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:19.916 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:19.915+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:20.014 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:35.931 100790== 2023-12-06T08:01:20.014 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:35.931 100790== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:01:20.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:20.015 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:20.016 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:20.016+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:20.025 INFO:tasks.ceph.mon.b.smithi107.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:01:20.055 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.922293895, 2023-12-06T08:01:20.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.057 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:20.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:20.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:20.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:20.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:20.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:20.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:20.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:20.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.45118044200001, 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.072 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:20.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:20.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:20.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:20.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.22098806299999, 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.086 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:20.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:20.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:20.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:20.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.220280514, 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:20.095 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:20.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:20.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:20.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.101 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.21964352000001, 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:20.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:20.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:20.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:20.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.162578325, 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.109 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:20.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:20.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:20.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.114 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.16108755400001, 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:20.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:20.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.128737892, 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.125 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.12210711700001, 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.130 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:20.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.12145643599999, 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.135 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.117983745, 2023-12-06T08:01:20.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:20.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.145 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.116845023, 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.11589732600001, 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.08753333199999, 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:20.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:20.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:20.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:20.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:20.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.059323815, 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:20.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:20.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.44912588099999, 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:20.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:20.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:20.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.44848104100001, 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.428014287, 2023-12-06T08:01:20.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:20.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.427265371, 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.183 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:20.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.426623302, 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:20.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:20.193 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:20.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:20.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:20.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:20.196 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:20.194+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:20.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:20.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:20.296 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:20.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:20.328 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:20.328 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:20.368 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:20.368 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:20.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:20.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:20.420 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:20.420 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:20.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:20.498 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:20.497+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:20.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:20.523 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:20.550 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:20.550 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:20.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:20.599 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:20.598+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:20.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:20.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:20.690 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:20.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:20.716 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:20.715+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:20.731 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:20.731 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:20.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:20.800 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:20.800+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:20.901 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:20.901+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:20.908 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:20.908 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:20.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:21.008 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:21.007+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:21.051 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.052 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:21.103 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:21.102+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:21.203 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:21.203+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:21.231 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.231 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:21.305 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:21.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:21.373 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.373 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:21.428 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:21.426+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:21.507 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:21.506+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:21.510 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.510 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:21.608 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:21.608+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:21.689 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.689 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:21.710 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:21.708+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:21.799 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:21.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:21.810 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:21.809+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:21.866 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:21.866 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:21.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:21.924 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:21.923+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:22.004 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.004 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.012 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:22.011+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:22.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:22.113 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:22.112+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:22.184 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.184 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:22.214 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:22.213+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:22.315 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:22.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:22.325 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.325 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:22.416 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:22.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:22.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T08:01:22.461 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.461 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:22.517 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:22.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:22.599 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.599 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:22.652 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:22.651+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:22.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:22.748 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:22.746+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:22.782 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.782 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:22.834 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:22.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:22.916 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:22.916 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:22.921 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:22.919+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:22.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:22.959 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:22.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:23.021 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:23.020+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:23.043 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:23.044 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:23.096 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:23.096 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:23.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:23.122 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:23.121+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:23.138 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:23.220 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:23.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:23.222 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:23.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:23.241 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:23.241 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:23.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:23.313 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:23.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:23.323 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:23.322+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:23.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:23.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:23.424 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:23.424+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:23.433 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:23.433 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:23.433 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:23.434 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:23.434 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:23.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:23.501 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:23.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:23.525 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:23.524+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:23.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:23.585 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:23.612 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:23.613 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:23.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:23.626 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:23.626+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:23.727 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:23.727+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:23.792 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:23.792 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:23.792 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:23.792 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:23.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:23.849 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:23.845+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:23.994 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:23.994 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:23.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:23.995 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:23.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:24.095 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:24.094+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:24.196 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:24.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:24.218 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 244.08826622800001, 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.219 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.61715277499999, 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.225 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.38696039600001, 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.231 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:24.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.38625284700001, 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.237 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:24.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:24.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:24.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:24.242 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.38561585299999, 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.32855065800001, 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:24.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.32705988699999, 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.254 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:24.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:24.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:24.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.259 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.29471022499999, 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.28807945, 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:24.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:24.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.287428769, 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.271 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:24.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.28395607799999, 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.276 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:24.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:24.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.28281735600001, 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.282 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.28186965899999, 2023-12-06T08:01:24.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:24.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:24.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.292 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.25350566500001, 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:24.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:24.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.22529614800001, 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.299 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 235.615098214, 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.305 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:24.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:24.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:24.310 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 235.61445337399999, 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:24.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.317 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 235.59398662000001, 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:24.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:24.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:24.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:24.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 235.59323770399999, 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:24.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:24.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:24.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:24.334 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 235.59259563500001, 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:24.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:24.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:24.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:24.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:24.341 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:24.342 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:24.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:24.344 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:24.343+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:24.445 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:24.444+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:24.472 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:24.472 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:24.472 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:24.472 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:24.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:24.565 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:24.564+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:24.647 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:24.647+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:24.649 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:24.650 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:24.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:24.748 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:24.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:24.829 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:24.830 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:24.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:24.849 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:24.848+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:24.949 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:24.949+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:24.971 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:24.971 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:24.972 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:24.972 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:24.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:25.050 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:25.049+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:25.108 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:25.108 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:25.108 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:25.108 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:25.108 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:25.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:25.151 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:25.151+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:25.252 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:25.251+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.87115649899999, 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:25.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:25.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.707657734000001, 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:25.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:25.297 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:25.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:25.353 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:25.352+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:25.439 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:25.439 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:25.439 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:25.439 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:25.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:25.454 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:25.453+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:01:25.501 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T08:01:25.502 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T08:01:25.502 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T08:01:25.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:25.556 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:25.554+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:25.576 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:25.576 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:25.576 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:25.576 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:25.576 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:25.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:25.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:25.656 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:25.655+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:25.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:25.713 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:25.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:25.713 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:25.713 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:25.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:25.763 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:25.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:25.857 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:25.856+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:25.893 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:25.893 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:25.893 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:25.893 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:25.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:25.958 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:25.958+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:26.030 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.030 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:26.030 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:26.030 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:26.030 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:26.059 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:26.058+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:26.117 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:26.171 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:26.171+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:26.251 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 245.269078124, 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:26.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.60529712300001, 2023-12-06T08:01:26.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:26.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.258 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:26.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:26.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:26.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:26.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:26.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:26.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.602016494, 2023-12-06T08:01:26.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:26.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:26.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.297239878, 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.296396781, 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.273861275, 2023-12-06T08:01:26.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:26.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:26.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.271868856, 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.287 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:26.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:26.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:26.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.27133659099999, 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:26.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.25441168200001, 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 244.25357504799999, 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.22912778, 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.22322541599999, 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:26.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:26.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:26.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.21851942000001, 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:26.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:26.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:26.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.21621754500001, 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.326 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.21571922199999, 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:26.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:26.330 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.215265311, 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:26.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 243.21404291100001, 2023-12-06T08:01:26.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 237.56887692999999, 2023-12-06T08:01:26.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 237.56143851499999, 2023-12-06T08:01:26.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:26.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:26.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 237.551365143, 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:26.347 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:26.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:26.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:26.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:26.351 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:26.353 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:26.352+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:26.453 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:26.453+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:26.520 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:26.521 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:26.557 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:26.556+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:26.658 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:26.657+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:26.698 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:26.698 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:26.793 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:26.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:26.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:26.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:26.801 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2023-12-06T08:01:26.801 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T08:01:26.801 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T08:01:26.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2023-12-06T08:01:26.859 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:26.859+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:26.893 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:26.894 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:26.894 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:26.952 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:26.953 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2023-12-06T08:01:26.953 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:26.960 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:26.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:27.062 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:27.062+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:27.076 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:27.076 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:27.076 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:27.076 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:27.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:27.162 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:27.162+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:27.214 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:27.214 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:27.214 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:27.214 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:27.214 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:27.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:27.264 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:27.264+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:27.364 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:27.364+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 226.81742177199999, 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:27.396 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:27.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:27.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:27.399 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.700244425999998, 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:27.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:27.403 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:27.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:27.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:27.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:27.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:27.404 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:27.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:27.465 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:27.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:27.538 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:27.538 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:27.538 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:27.539 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:27.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:27.588 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:27.587+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:27.667 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:27.667+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:27.715 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:27.715 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:27.715 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:27.716 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:27.716 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:27.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:27.768 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:27.767+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:27.851 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:27.851 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:27.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:27.906 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:27.906+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:27.969 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:27.969+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:27.991 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:27.991 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:27.991 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:27.991 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:27.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:28.086 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:28.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:28.168 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:28.169 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:28.172 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:28.171+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:28.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:28.272 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:28.272+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:28.400 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:28.400 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:28.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:28.400 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:28.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 248.270257547, 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:28.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:28.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:28.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.79914409400001, 2023-12-06T08:01:28.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:28.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.416 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.568951715, 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.417 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.423 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.568244166, 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:28.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:28.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.56760717200001, 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:28.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:28.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:28.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:28.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.510541977, 2023-12-06T08:01:28.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:28.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:28.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.50905120600001, 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:28.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:28.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.47670154400001, 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:28.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:28.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:28.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:28.458 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.47007076899999, 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:28.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:28.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:28.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:28.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.46942008799999, 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 245.46594739700001, 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:28.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:28.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:28.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 245.464808675, 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:28.482 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 245.46386097800001, 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:28.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 245.435496984, 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:28.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:28.495 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 245.407287467, 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:28.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:28.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:28.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:28.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:28.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.79708953299999, 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:28.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.79644469300001, 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:28.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:28.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.775977939, 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.514 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.77522902300001, 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 239.774586954, 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:28.533 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:28.534 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:28.399+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:28.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:28.536 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:28.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:28.638 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:28.638+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:28.704 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:28.704 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:28.704 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:28.704 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:28.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:28.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T08:01:28.758 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:28.758+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:28.840 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:28.840+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:28.848 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:28.849 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:28.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:28.942 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:28.942+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:29.030 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:29.030 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:29.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:29.030 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:29.030 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:29.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:29.080 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:29.080+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:29.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:01:29.175 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:29.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:29.212 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:29.212 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:29.212 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:29.212 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:29.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:29.244 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:29.243+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:29.347 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:29.347+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:29.393 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:29.393 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:29.393 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:29.394 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:29.394 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:29.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:29.488 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:29.487+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:29.565 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:29.565+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:29.597 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 230.17546593700001, 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:29.598 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:29.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 50.011967171999999, 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:29.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:29.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:29.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:29.606 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:29.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:29.650 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:29.650+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:29.749 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:29.749 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:29.749 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:29.749 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:29.750 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:29.750+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:29.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:29.851 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:29.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:29.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:29.981 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:29.981+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:30.053 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:30.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:30.064 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.064 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:30.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:30.064 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:30.064 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:30.154 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:30.154+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:30.243 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:30.243 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:30.255 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:30.255+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:30.357 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:30.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:30.380 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.381 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:30.381 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:30.381 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:30.381 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:30.458 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:30.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:30.557 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.557 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:30.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 249.57658555, 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:30.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:30.562 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.91280454899999, 2023-12-06T08:01:30.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:30.563 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.563 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.563 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:30.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:30.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:30.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:30.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:30.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:30.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:30.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:30.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:30.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:30.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.575 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.90952392, 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:30.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:30.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.604747304, 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.603904207, 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:30.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:30.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.581368701, 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:30.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.579376282, 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.57884401699999, 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.596 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:30.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.56191910800001, 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 248.56108247399999, 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:30.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.536635206, 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:30.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:30.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:30.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.611 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.53073284199999, 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.52602684600001, 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:30.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:30.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:30.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.52372497100001, 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:30.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:30.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:30.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:30.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.52322664799999, 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.624 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.522772737, 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:30.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:30.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 247.52155033700001, 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:30.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:30.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:30.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:30.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 241.87638435599999, 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:30.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:30.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.640 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 241.86894594099999, 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:30.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.644 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 241.858872569, 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:30.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:30.648 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:30.649 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:30.650 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:30.650+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:30.752 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:30.751+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:30.817 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.818 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:30.818 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:30.818 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:30.855 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:30.853+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:30.953 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:30.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:30.957 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:30.957 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:30.958 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:30.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:31.054 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:31.054+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:31.138 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:31.138 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:31.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:31.139 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:31.139 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:31.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:31.155 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:31.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:31.256 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:31.256+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:31.276 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:31.276 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:31.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:31.357 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:31.356+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:31.417 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:31.417 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:31.417 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:31.417 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:31.417 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:31.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:31.458 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:31.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:31.558 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:31.558+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.01987330599999, 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 50.902695960000003, 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:31.607 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:31.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:31.659 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:31.659+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:31.751 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:31.752 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:31.752 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:31.752 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:31.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:31.762 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:31.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:31.864 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:31.863+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:31.927 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:31.928 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:31.928 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:31.928 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:31.928 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:31.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:31.962 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:31.962 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:31.963 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2023-12-06T08:01:31.963 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T08:01:31.963 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T08:01:31.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2023-12-06T08:01:31.980 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:31.980+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:32.064 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:32.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:32.066 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:32.066 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:32.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:32.066 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:32.066 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:32.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:32.093 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:32.093 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "" 2023-12-06T08:01:32.093 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:32.166 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:32.165+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:32.249 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:32.249 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:32.249 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:32.249 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:32.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:32.303 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:32.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:32.367 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:32.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:32.387 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:32.387 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:32.387 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:32.387 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:32.387 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:32.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:32.481 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:32.481+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:32.569 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:32.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:32.620 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.49051720899999, 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 251.019403756, 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.78921137699999, 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.78850382799999, 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.787866834, 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.73080163899999, 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.729310868, 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.696961206, 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.69033043100001, 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 250.68967975000001, 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:32.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 249.686207059, 2023-12-06T08:01:32.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:32.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:32.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:32.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 249.68506833699999, 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.704 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:32.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:32.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:32.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 249.68412064, 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.709 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.714 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 249.65575664599999, 2023-12-06T08:01:32.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:32.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:32.722 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 249.62754712899999, 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.723 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 244.01734919500001, 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:32.728 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:32.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:32.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:32.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 244.016704355, 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.734 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:32.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:32.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:32.739 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 243.99623760099999, 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:32.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 243.995488685, 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:32.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:32.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 243.99484661599999, 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:32.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:32.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:32.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:32.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.755 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:32.755 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:32.755 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:32.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:32.757 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:32.756+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:32.857 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:32.857+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:32.925 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:32.925 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:32.925 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:32.925 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:32.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:32.958 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:32.958+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:33.062 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:33.062 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:33.062 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:33.062 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:33.062 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:33.067 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:33.063+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:33.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:33.161 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:33.161+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:33.200 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:33.200 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:33.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:33.262 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:33.262+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:33.363 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:33.363+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:33.380 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:33.380 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:33.380 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:33.380 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:33.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:33.464 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:33.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:33.517 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:33.518 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:33.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:33.569 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:33.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:33.666 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:33.666+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:33.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 234.27945755100001, 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:33.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:33.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:33.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.115958786, 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:33.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:33.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:33.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:33.707 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:33.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:33.767 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:33.766+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:33.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:33.903 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:33.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:33.967 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:33.967+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:34.030 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.030 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:34.030 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:34.030 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:34.030 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:34.069 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:34.069+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:34.169 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.169 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:34.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:34.169 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:34.169 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.170 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:34.169+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:34.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:34.271 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:34.270+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:34.348 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.348 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:34.348 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:34.348 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:34.401 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:34.401+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:34.473 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:34.472+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:34.527 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.527 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:34.527 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:34.527 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:34.527 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:34.580 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:34.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:34.626 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:01:34.641 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T08:01:34.641 INFO:tasks.mon_recovery:causing some monitor log activity 2023-12-06T08:01:34.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2023-12-06T08:01:34.677 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:34.676+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:34.705 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 253.72422375100001, 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 253.06044274999999, 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:34.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 253.057162121, 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:34.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:34.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.75238550500001, 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:34.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.75154240800001, 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:34.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.72900690200001, 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:34.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:34.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:34.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.727014483, 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.735 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:34.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.726482218, 2023-12-06T08:01:34.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:34.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:34.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.70955730899999, 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.743 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:34.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.708720675, 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.68427340700001, 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:34.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.678371043, 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:34.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:34.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.67366504699999, 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.762 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.67136317200001, 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:34.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.670864849, 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:34.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:34.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:34.770 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.670410938, 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:34.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:34.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 251.66918853799999, 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:34.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 246.024022557, 2023-12-06T08:01:34.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:34.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.782 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 246.016584142, 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:34.786 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 246.00651077000001, 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:34.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:34.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:34.790 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:34.792 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:34.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:34.893 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:34.893+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:34.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:34.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:34.961 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:34.961 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:34.961 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:34.962 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:34.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:34.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:34.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:34.996 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:34.996+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:35.092 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:35.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:35.095 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:35.095+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:35.099 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:35.099 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:35.100 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:35.100 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:35.100 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:35.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:35.179 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:35.179 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:35.195 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:35.194+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:35.236 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:35.236 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:35.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:35.236 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:35.237 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:35.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:35.267 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:35.267 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:35.331 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:35.330+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:35.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:35.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:35.397 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:35.397+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:35.417 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:35.417 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:35.417 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:35.417 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:35.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:35.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:35.449 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:35.536 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:35.537 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:35.537 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:35.536+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:35.596 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:35.597 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:35.604 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:35.604+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:35.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:35.705 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:35.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:35.777 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:35.777 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:35.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:01:35.805 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:35.805+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:35.910 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:35.910+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:35.926 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:35.926 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:35.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:36.012 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:36.010+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:36.066 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.066 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:36.112 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:36.111+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:36.229 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:36.228+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:36.243 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.244 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:36.317 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:36.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:36.415 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:36.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:36.423 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.423 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:36.517 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:36.517+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:36.561 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.561 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:36.618 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:36.618+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:36.699 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.699 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:36.720 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:36.719+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:36.821 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:36.820+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:36.843 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:36.843 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:36.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:36.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2023-12-06T08:01:36.921 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:36.921+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:37.034 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:37.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:37.052 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.052 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:37.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:37.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:37.104 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-12-06T08:01:37.104 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T08:01:37.104 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T08:01:37.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-12-06T08:01:37.151 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:37.150+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:37.231 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:37.231+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:37.246 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "" 2023-12-06T08:01:37.246 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:37.246 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.246 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:37.326 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:37.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:37.389 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.389 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:37.444 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:37.440+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:37.529 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:37.528+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:37.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:37.581 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.581 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:37.631 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:37.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:37.730 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:37.729+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:37.777 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.777 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:37.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:37.818 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:37.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:01:37.832 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:37.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:37.915 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:37.915 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:37.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:37.965 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:37.965+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:37.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2023-12-06T08:01:38.034 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:38.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:38.093 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:38.093 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:38.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:38.110 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:38.110 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:38.148 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:38.143+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:38.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:38.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:38.237 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:38.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:38.274 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:38.274 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:38.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:38.285 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:38.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:38.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:38.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:38.378 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:38.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:38.441 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:38.441 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:38.442 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:38.441+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:38.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:38.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:38.499 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:38.542 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:38.542+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:38.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:38.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:38.632 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:38.632 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:38.633 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:38.633 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:38.633 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:38.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:38.646 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:38.646+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:38.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:38.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:38.746 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:38.746+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:38.847 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:38.847+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:38.851 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:38.851 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 257.871152903, 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:38.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:38.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:38.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.856 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 257.20737190199998, 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:38.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 257.20409127300002, 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:38.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:38.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.89931465699999, 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:38.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:38.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:38.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.89847156000002, 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.87593605400002, 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.87394363499999, 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.87341136999999, 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.85648646099997, 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:38.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.895 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.85564982699998, 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:38.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:38.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.899 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.83120255899999, 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:38.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:38.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.903 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.82530019500001, 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:38.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:38.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.820594199, 2023-12-06T08:01:38.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.818292324, 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:38.916 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.81779400100001, 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:38.920 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.81734008999999, 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:38.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:38.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.925 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 255.81611769, 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:38.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.930 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.17095170900001, 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:38.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:38.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:38.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.934 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.16351329400001, 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:38.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:38.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:38.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:38.940 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.15343992199999, 2023-12-06T08:01:38.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:38.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:38.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:38.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:38.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:38.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:38.944 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:38.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:38.949 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:38.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:38.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2023-12-06T08:01:39.002 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:01:39.057 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:39.057+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:39.113 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:39.114 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:39.150 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:39.150+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:39.251 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:39.251 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.252 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:39.251+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:39.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:39.353 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:39.352+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:39.387 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.387 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:39.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:39.387 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:39.387 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:39.454 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:39.453+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:39.555 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:39.555+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:39.564 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.564 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:39.564 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:39.565 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:39.658 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:39.658+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:39.701 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:39.701 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:39.701 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:39.759 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:39.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 239.26488034900001, 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:39.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:39.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.147703002999997, 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:39.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:39.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:39.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:39.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:39.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:39.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:39.854 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:39.854 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:39.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:39.898 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:39.897+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:39.962 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:39.961+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:39.996 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:39.996 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:39.996 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:39.997 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2023-12-06T08:01:40.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:40.065 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:40.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:40.165 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:40.165+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:40.179 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:40.180 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:40.180 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:40.180 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:40.180 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:40.267 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:40.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:40.358 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:40.358 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:40.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:40.358 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:40.358 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:40.368 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:40.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:40.468 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:40.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:40.511 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:40.511 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:40.511 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:40.512 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:40.569 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:40.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:40.653 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:40.653 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:40.653 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:40.653 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:40.653 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:40.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:40.705 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:40.705+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:40.772 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:40.772+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:40.859 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 260.72047054000001, 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:40.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:40.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 259.24935708700002, 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:40.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:40.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:40.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:40.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 259.01916470800001, 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:40.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:40.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:40.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:40.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:40.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 259.01845715899998, 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.881 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:40.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:40.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 259.01782016499999, 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.886 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:40.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:40.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:40.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.96075496999998, 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.95926419900002, 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.902 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.92691453700002, 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:40.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:40.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:40.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:40.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.908 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.920283762, 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:40.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:40.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.91963308099997, 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.914 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.91616039000002, 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:40.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.924 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.91502166800001, 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:40.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:40.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:40.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.91407397099999, 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.930 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:40.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:40.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.935 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.88570997699998, 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.936 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:40.941 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.85750045999998, 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.247302526, 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.947 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:40.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:40.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.24665768599999, 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.952 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:40.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:40.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:40.958 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.22619093200001, 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:40.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:40.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:40.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:40.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:40.963 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.22544201599999, 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.964 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 252.22479994700001, 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:40.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:40.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:40.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:40.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:40.974 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:40.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:40.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:40.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:40.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2023-12-06T08:01:40.978 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:40.976+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:41.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:01:41.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:41.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:41.076 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:41.076+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:41.177 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:41.177+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:41.204 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:41.205 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:41.205 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:41.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:41.278 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:41.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:41.299 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:41.299 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:41.366 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:41.366 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:41.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:41.386 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:41.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:41.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:41.387 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:41.480 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:41.480+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:41.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:41.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:41.550 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:41.551 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:41.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:41.589 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:41.588+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:41.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:41.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:41.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:41.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:41.682 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:41.682+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:41.720 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:41.720 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:41.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:41.783 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:41.783+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:41.884 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:41.884+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:41.899 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:41.899 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:41.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:41.985 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:41.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:42.063 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.063 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2023-12-06T08:01:42.074 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:01:42.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:42.089 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:42.089+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:42.187 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:42.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:42.208 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.208 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:42.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:42.258 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:42.258 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-06T08:01:42.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-06T08:01:42.303 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:42.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:42.385 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.385 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.389 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:42.389+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:42.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:42.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-06T08:01:42.489 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:42.489+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:42.574 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.574 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:42.625 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:42.624+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:42.693 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:42.691+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:42.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-06T08:01:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.752 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:42.805 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:42.803+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:42.894 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:42.894+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:42.931 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:42.931 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:42.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:42.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-06T08:01:43.031 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:43.031+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:43.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2023-12-06T08:01:43.071 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.071 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:43.125 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:43.124+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:43.197 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:43.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:43.209 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.209 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2023-12-06T08:01:43.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:43.303 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:43.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:43.389 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.389 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:43.399 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:43.398+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:43.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2023-12-06T08:01:43.514 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:43.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:43.567 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.567 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:43.622 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:43.621+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:43.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:43.701 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:43.701+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:43.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2023-12-06T08:01:43.803 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:43.803+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:43.884 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:43.884 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:43.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:43.927 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:43.927 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:43.928 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:43.923+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:43.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2023-12-06T08:01:44.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:44.012 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:44.013 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:44.012+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:44.064 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:44.064 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:44.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:44.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:01:44.105 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:44.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:44.106 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:44.106+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:44.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2023-12-06T08:01:44.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:44.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:44.228 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:44.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:44.239 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:44.239 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:44.239 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:44.239 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:44.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:44.289 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:40.722 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.300 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:40.732 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.307 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:44.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:44.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:44.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:44.325 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:00.236 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.342 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:00:40.774 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.367 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:00.279 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.408 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:44.407+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:44.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:44.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:44.435 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:44.435 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:44.436 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:44.436 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:44.436 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:44.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:44.503 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:44.503 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:44.509 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:44.508+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:44.604 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:44.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:44.619 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:44.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:44.633 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:44.633 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:44.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:44.710 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:00.621 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:01:44.710 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:44.710+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:44.811 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:44.811+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:44.825 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:44.826 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:44.826 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:44.826 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:44.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:44.921 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:44.921+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:45.014 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:45.014+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:45.042 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:45.042 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:45.042 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:45.042 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:45.042 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:45.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:45.139 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:45.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:45.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2023-12-06T08:01:45.216 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:45.216+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:45.294 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 265.15733492300001, 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:45.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:45.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:45.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:45.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.68622147000002, 2023-12-06T08:01:45.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:45.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:45.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.306 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.456029091, 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.45532154199998, 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.312 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:45.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:45.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.45468454799999, 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:45.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.39761935299998, 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:45.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:45.329 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.39612858200002, 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:45.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.36377892000002, 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.335 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:45.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:45.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.357148145, 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:45.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 263.35649746399997, 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.347 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:45.352 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 262.35302477300002, 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.353 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:45.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:45.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.360 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 262.35188605100001, 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:45.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:45.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:45.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:45.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 262.35093835399999, 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.366 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:45.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:45.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:45.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:45.371 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 262.32257435999998, 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:45.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:45.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:45.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:45.377 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 262.29436484299998, 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:45.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.684166909, 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:45.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:45.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.68352206899999, 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.388 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:45.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:45.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.66305531500001, 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.394 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:45.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:45.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:45.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:45.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.66230639899999, 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.66166433000001, 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:45.405 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:45.410 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:45.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:45.412 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:45.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:45.513 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:45.513+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:45.581 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:45.582 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:45.582 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:45.582 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:45.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:45.633 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:45.633+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:45.721 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:45.721+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:45.765 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:45.766 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:45.766 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:45.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:01:45.815 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:45.815+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:45.916 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:45.916 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:45.916 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:45.916+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:45.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:46.017 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:46.017+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:46.092 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:46.092 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:46.092 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:46.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:46.118 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:46.118+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:46.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2023-12-06T08:01:46.244 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:46.244+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:46.274 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:46.274 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:46.274 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:46.274 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:46.274 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:46.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:46.326 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:46.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:46.420 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:46.420+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:46.464 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 247.04230600899999, 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:46.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:46.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:46.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:46.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.878807244000001, 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:46.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:46.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:46.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 6.8717462300000003, 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:46.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:46.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:01:46.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:01:46.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:46.479 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:46.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:46.522 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:46.522+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:46.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:46.620 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:46.620 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:46.620 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:46.620 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:46.622 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:46.621+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:46.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:46.742 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:46.741+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:46.756 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:46.756 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:46.756 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:46.756 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:46.756 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:46.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:46.851 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:46.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:46.898 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:46.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:46.924 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:46.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:46.935 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:46.936 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:46.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:46.936 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:46.936 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:46.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:46.981 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:47.025 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:47.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:47.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:47.064 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:47.125 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:47.125+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:47.128 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:47.129 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:47.129 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:47.129 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:47.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:47.159 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:47.159 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:47.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2023-12-06T08:01:47.239 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:47.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:47.261 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:47.262 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:47.329 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:47.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:47.356 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:47.356 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:47.357 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:47.357 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:47.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:47.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:47.449 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:47.452 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:47.452+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:47.544 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:47.543+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:47.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:47.568 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:47.571 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:47.571 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:47.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:47.664 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:47.664+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:47.735 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:47.735+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:47.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:47.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:01:47.844 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:47.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:47.928 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:47.928 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:47.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:47.939 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:47.937+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:48.038 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:48.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:48.106 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.106 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:48.139 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:48.139+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:48.240 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:48.240+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:48.243 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.243 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:48.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2023-12-06T08:01:48.350 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:48.350+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:48.380 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.380 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:48.443 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:48.443+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:48.517 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.517 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:48.568 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:48.567+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:48.645 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:48.645+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:48.654 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.655 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:48.748 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:48.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:48.834 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:48.834 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:48.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:48.847 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:48.847+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:48.948 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:48.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:49.013 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.013 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:49.068 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:49.068+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:49.150 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:49.151+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:49.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.152 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:49.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2023-12-06T08:01:49.252 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:49.252+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:49.353 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:49.353+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:49.362 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:49.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:49.362 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.362 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:49.454 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:49.454+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:49.467 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:01:49.509 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.510 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:49.555 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:49.555+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:49.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:49.656 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:49.656+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:49.709 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.709 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:49.758 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:49.758+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:49.860 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:49.859+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:49.860 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:49.860 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:49.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:49.867 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:49.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:49.961 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:49.961+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:49.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:50.039 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:50.039 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:50.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:50.093 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:50.093+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:50.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:50.120 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:50.164 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:50.163+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:50.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:50.241 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:50.241 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:50.242 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:50.242 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:50.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:50.266 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:50.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:50.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2023-12-06T08:01:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:50.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:50.366 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:50.366+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:50.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:50.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:50.468 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:50.467+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:50.486 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:50.486 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:50.487 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:50.487 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:50.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:50.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:50.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:50.569 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:50.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:50.629 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:50.629 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:50.653 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 251.232431158, 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:50.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:50.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:50.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:50.658 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.068932392999997, 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:50.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:50.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.061871378999999, 2023-12-06T08:01:50.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:50.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:01:50.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:50.666 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:50.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:50.709 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:50.709+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:50.771 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:50.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:50.797 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:50.797 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:50.797 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:50.797 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:50.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:50.872 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:50.872+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:50.934 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:50.934 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:50.934 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:50.935 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:50.935 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:50.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:50.986 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:50.986+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:51.074 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:51.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:51.114 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:51.114 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:51.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:51.114 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:51.114 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:51.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:51.174 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:51.174+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:51.250 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:51.250 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:51.251 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:51.251 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:51.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:51.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2023-12-06T08:01:51.301 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:51.302+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:51.376 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:51.376+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:51.428 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:51.429 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:51.429 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:51.429 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:51.429 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:51.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:51.481 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:51.480+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:51.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:01:51.578 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:51.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:51.645 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 270.66542068000001, 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:01:51.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:01:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:01:51.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 270.00163967899999, 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:01:51.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:01:51.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:01:51.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.99835904999998, 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:01:51.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.69358243400001, 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:01:51.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:01:51.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.69273933699998, 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.67020383099998, 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:01:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:01:51.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:01:51.669 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.66821141200001, 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:01:51.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:51.673 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.667679147, 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.65075423799999, 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:01:51.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:01:51.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.649917604, 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:01:51.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.62547033599998, 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:01:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.61956797200003, 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:01:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.61486197599999, 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:01:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:01:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.61256010099999, 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.612061778, 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:01:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:01:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.61160786699998, 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:01:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 268.61038546700001, 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:01:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:01:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 262.96521948600002, 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:01:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 262.957781071, 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:01:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:01:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 262.94770769899998, 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:51.728 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:51.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:51.729 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:51.729+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:51.831 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:51.830+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:51.905 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:51.906 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:51.906 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:51.906 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:51.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:51.932 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:51.932+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:52.033 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:52.032+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:52.043 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:52.043 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:52.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:52.134 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:52.134+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:52.182 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:01:52.182 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:52.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:52.235 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:52.234+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:52.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2023-12-06T08:01:52.322 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:01:52.323 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:52.327 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:01:52.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:52.377 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:52.376+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:52.436 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:52.436+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:52.460 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:52.460 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:52.460 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:52.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:52.537 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:52.537+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:52.638 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:52.638+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.06268145199999, 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 71.945504106000001, 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:52.647 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.938699890000001, 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:01:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:01:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:52.655 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:52.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:52.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:52.739 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:52.739+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:52.802 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:52.802 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:52.802 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:52.802 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:52.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:52.854 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:52.853+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:52.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:52.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:52.941 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:52.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:52.988 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:52.988 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:52.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:53.042 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:53.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:53.043 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:53.043 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:53.129 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:53.130 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:53.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:53.131 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:53.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:53.143 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:53.142+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:53.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:53.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:53.243 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:53.242+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:53.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2023-12-06T08:01:53.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:53.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:53.322 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:53.322 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:53.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:53.374 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:53.374+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:53.399 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:53.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:53.444 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:53.444+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:53.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:53.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:53.527 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:53.527 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:53.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:53.578 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:53.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:53.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:53.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:53.649 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:53.649+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:53.711 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:53.711 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:53.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:53.751 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:53.751+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:53.851 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:53.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:53.896 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:53.897 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:53.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:53.954 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:53.954+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:54.055 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:54.055+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:54.075 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.075 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:54.158 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:54.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:54.256 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.256 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.257 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:54.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:54.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:54.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2023-12-06T08:01:54.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:01:54.362 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:01:54.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:01:54.363 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-12-06T08:01:54.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-12-06T08:01:54.364 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:54.363+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:54.459 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:54.460+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:54.483 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.483 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:54.561 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:54.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:54.631 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.631 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:54.679 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:54.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:54.763 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:54.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:54.832 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.832 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:54.892 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:54.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:54.965 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:54.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:54.981 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:54.981 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:54.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:55.075 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:55.075+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:55.160 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:55.160 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:55.167 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:55.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:55.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:55.267 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:55.268+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:55.338 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:55.338 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:55.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2023-12-06T08:01:55.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:55.388 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:55.388+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 64, 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "aba07bce-2203-46e5-bd3f-65b300a2f9db", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2023-12-06T07:56:32.091884+0000", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2023-12-06T08:01:52.223654+0000", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2023-12-06T07:57:18.326272+0000", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2023-12-06T07:56:37.562649+0000", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2023-12-06T08:01:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 6, 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2023-12-06T08:01:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "reef", 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:18.316796+0000", 2023-12-06T08:01:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2023-12-06T08:01:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T08:01:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "18", 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T08:01:55.462 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T08:01:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T08:01:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2023-12-06T08:01:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 8, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 8, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.25, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "rbd", 2023-12-06T08:01:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2023-12-06T07:57:26.381263+0000", 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 1, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 8, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 8, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 8, 2023-12-06T08:01:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 8, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 8, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "22", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2023-12-06T08:01:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-12-06T08:01:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2023-12-06T08:01:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9900000095367432, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9900000095367432, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2023-12-06T08:01:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "0da60772-f74a-424c-b41a-462ed8a484ac", 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6800", 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2023-12-06T08:01:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2023-12-06T08:01:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1405545511 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6800/1405545511", 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6801/1405545511", 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6803/1405545511", 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6802/1405545511", 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "aa3bcc32-a99b-4382-b6d8-a9561dc53073", 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 56, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1973529083 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6812/1973529083", 2023-12-06T08:01:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/1973529083", 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6815/1973529083", 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6814/1973529083", 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea5cc880-085f-4d6a-9307-212ab3aee1ce", 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1651824594 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6804/1651824594", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1651824594", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6807/1651824594", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6806/1651824594", 2023-12-06T08:01:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "f3f7cd68-c256-4e03-9a53-5a88e9e19c12", 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 16, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 51, 2023-12-06T08:01:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3951389031 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6808/3951389031", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6809/3951389031", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6811/3951389031", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6810/3951389031", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d078280f-f9dc-490e-a13f-4714b9d70ca8", 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 15, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 45, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6805", 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6806", 2023-12-06T08:01:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6808", 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6807", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2922507264 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6805/2922507264", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6806/2922507264", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6808/2922507264", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6807/2922507264", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "03c3b27d-6041-413a-a1d5-807f1349decb", 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 19, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6809", 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6810", 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6812", 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6811", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 933623275 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6809/933623275", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6810/933623275", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6812/933623275", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6811/933623275", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "332ffee8-2eca-481a-a2a4-ca5edfbebfe2", 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0.1891021728515625, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 14, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6813", 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6814", 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6816", 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6815", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 584359350 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6813/584359350", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6814/584359350", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6816/584359350", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6815/584359350", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "6eab830f-d790-454c-9c56-5092fab218c4", 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 24, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6801", 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6802", 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6804", 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.107:6803", 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1873215283 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.107:6801/1873215283", 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.107:6802/1873215283", 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.107:6804/1873215283", 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.107:6803/1873215283", 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2023-12-06T08:01:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.731245+0000", 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.384135+0000", 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.655702+0000", 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:15.515341+0000", 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.584483+0000", 2023-12-06T08:01:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.330101+0000", 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:14.433563+0000", 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2023-12-06T08:01:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540138322906710015, 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2023-12-06T07:57:13.629220+0000", 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.0", 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: 4, 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.5", 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: 6 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: 2 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.7", 2023-12-06T08:01:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: 7 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.6", 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 1 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": {}, 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2023-12-06T08:01:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:55.516 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:55.518 INFO:tasks.thrashosds.thrasher:j is {'epoch': 64, 'fsid': 'aba07bce-2203-46e5-bd3f-65b300a2f9db', 'created': '2023-12-06T07:56:32.091884+0000', 'modified': '2023-12-06T08:01:52.223654+0000', 'last_up_change': '2023-12-06T07:57:18.326272+0000', 'last_in_change': '2023-12-06T07:56:37.562649+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-06T07:57:18.316796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-06T07:57:26.381263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0da60772-f74a-424c-b41a-462ed8a484ac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6800', 'nonce': 1405545511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 1405545511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 1405545511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1405545511}]}, 'public_addr': '172.21.15.84:6800/1405545511', 'cluster_addr': '172.21.15.84:6801/1405545511', 'heartbeat_back_addr': '172.21.15.84:6803/1405545511', 'heartbeat_front_addr': '172.21.15.84:6802/1405545511', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aa3bcc32-a99b-4382-b6d8-a9561dc53073', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 1973529083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 1973529083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 1973529083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 1973529083}]}, 'public_addr': '172.21.15.84:6812/1973529083', 'cluster_addr': '172.21.15.84:6813/1973529083', 'heartbeat_back_addr': '172.21.15.84:6815/1973529083', 'heartbeat_front_addr': '172.21.15.84:6814/1973529083', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ea5cc880-085f-4d6a-9307-212ab3aee1ce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1651824594}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 1651824594}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 1651824594}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1651824594}]}, 'public_addr': '172.21.15.84:6804/1651824594', 'cluster_addr': '172.21.15.84:6805/1651824594', 'heartbeat_back_addr': '172.21.15.84:6807/1651824594', 'heartbeat_front_addr': '172.21.15.84:6806/1651824594', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3f7cd68-c256-4e03-9a53-5a88e9e19c12', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 3951389031}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3951389031}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3951389031}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3951389031}]}, 'public_addr': '172.21.15.84:6808/3951389031', 'cluster_addr': '172.21.15.84:6809/3951389031', 'heartbeat_back_addr': '172.21.15.84:6811/3951389031', 'heartbeat_front_addr': '172.21.15.84:6810/3951389031', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd078280f-f9dc-490e-a13f-4714b9d70ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6805', 'nonce': 2922507264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 2922507264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 2922507264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6807', 'nonce': 2922507264}]}, 'public_addr': '172.21.15.107:6805/2922507264', 'cluster_addr': '172.21.15.107:6806/2922507264', 'heartbeat_back_addr': '172.21.15.107:6808/2922507264', 'heartbeat_front_addr': '172.21.15.107:6807/2922507264', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03c3b27d-6041-413a-a1d5-807f1349decb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6809', 'nonce': 933623275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 933623275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 933623275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6811', 'nonce': 933623275}]}, 'public_addr': '172.21.15.107:6809/933623275', 'cluster_addr': '172.21.15.107:6810/933623275', 'heartbeat_back_addr': '172.21.15.107:6812/933623275', 'heartbeat_front_addr': '172.21.15.107:6811/933623275', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '332ffee8-2eca-481a-a2a4-ca5edfbebfe2', 'up': 1, 'in': 1, 'weight': 0.1891021728515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6813', 'nonce': 584359350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 584359350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 584359350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6815', 'nonce': 584359350}]}, 'public_addr': '172.21.15.107:6813/584359350', 'cluster_addr': '172.21.15.107:6814/584359350', 'heartbeat_back_addr': '172.21.15.107:6816/584359350', 'heartbeat_front_addr': '172.21.15.107:6815/584359350', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6eab830f-d790-454c-9c56-5092fab218c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6801', 'nonce': 1873215283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 1873215283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 1873215283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6803', 'nonce': 1873215283}]}, 'public_addr': '172.21.15.107:6801/1873215283', 'cluster_addr': '172.21.15.107:6802/1873215283', 'heartbeat_back_addr': '172.21.15.107:6804/1873215283', 'heartbeat_front_addr': '172.21.15.107:6803/1873215283', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.731245+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.384135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.655702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:15.515341+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.584483+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.330101+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:14.433563+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-12-06T07:57:13.629220+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [4, 3]}, {'pgid': '2.5', 'osds': [1, 6]}, {'pgid': '2.6', 'osds': [7, 2]}, {'pgid': '2.7', 'osds': [1, 7]}], 'pg_upmap_items': [{'pgid': '2.6', 'mappings': [{'from': 0, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-12-06T08:01:55.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:01:55.519 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:01:55.520 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:55.518+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:55.520 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:55.520 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:55.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:01:55.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:01:55.619 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:55.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:55.699 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:55.699 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:55.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:01:55.753 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:55.753+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:55.820 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:55.821+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:55.880 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:55.880 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:55.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:01:55.921 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:55.921+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:56.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:56.009 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:56.043 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:56.044+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:56.058 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:56.058 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:56.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:01:56.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:56.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:56.123 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:56.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:56.163 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:01:56.164 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:01:56.174 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2023-12-06T08:01:55.848275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222180,"kb_used_data":6016,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752540,"statfs":{"total":773094113280,"available":772866600960,"internally_reserved":0,"allocated":6160384,"data_stored":2770076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.431223"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-06T08:01:50.340067+0000","last_change":"2023-12-06T08:01:15.480948+0000","last_active":"2023-12-06T08:01:50.340067+0000","last_peered":"2023-12-06T08:01:50.340067+0000","last_clean":"2023-12-06T08:01:50.340067+0000","last_became_active":"2023-12-06T08:01:15.474602+0000","last_became_peered":"2023-12-06T08:01:15.474602+0000","last_unstale":"2023-12-06T08:01:50.340067+0000","last_undegraded":"2023-12-06T08:01:50.340067+0000","last_fullsized":"2023-12-06T08:01:50.340067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:07:38.010954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-12-06T08:01:55.057487+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T08:01:55.057487+0000","last_peered":"2023-12-06T08:01:55.057487+0000","last_clean":"2023-12-06T08:01:55.057487+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T08:01:55.057487+0000","last_undegraded":"2023-12-06T08:01:55.057487+0000","last_fullsized":"2023-12-06T08:01:55.057487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-06T08:01:50.337398+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:01:50.337398+0000","last_peered":"2023-12-06T08:01:50.337398+0000","last_clean":"2023-12-06T08:01:50.337398+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:01:50.337398+0000","last_undegraded":"2023-12-06T08:01:50.337398+0000","last_fullsized":"2023-12-06T08:01:50.337398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-06T08:01:46.300068+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T08:01:46.300068+0000","last_peered":"2023-12-06T08:01:46.300068+0000","last_clean":"2023-12-06T08:01:46.300068+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T08:01:46.300068+0000","last_undegraded":"2023-12-06T08:01:46.300068+0000","last_fullsized":"2023-12-06T08:01:46.300068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-06T08:01:45.778807+0000","last_change":"2023-12-06T08:01:04.450104+0000","last_active":"2023-12-06T08:01:45.778807+0000","last_peered":"2023-12-06T08:01:45.778807+0000","last_clean":"2023-12-06T08:01:45.778807+0000","last_became_active":"2023-12-06T08:01:04.441340+0000","last_became_peered":"2023-12-06T08:01:04.441340+0000","last_unstale":"2023-12-06T08:01:45.778807+0000","last_undegraded":"2023-12-06T08:01:45.778807+0000","last_fullsized":"2023-12-06T08:01:45.778807+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:26:50.232248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-12-06T08:01:55.051335+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T08:01:55.051335+0000","last_peered":"2023-12-06T08:01:55.051335+0000","last_clean":"2023-12-06T08:01:55.051335+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T08:01:55.051335+0000","last_undegraded":"2023-12-06T08:01:55.051335+0000","last_fullsized":"2023-12-06T08:01:55.051335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-06T08:01:47.271947+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:01:47.271947+0000","last_peered":"2023-12-06T08:01:47.271947+0000","last_clean":"2023-12-06T08:01:47.271947+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:01:47.271947+0000","last_undegraded":"2023-12-06T08:01:47.271947+0000","last_fullsized":"2023-12-06T08:01:47.271947+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-12-06T08:01:45.782676+0000","last_change":"2023-12-06T07:57:31.133661+0000","last_active":"2023-12-06T08:01:45.782676+0000","last_peered":"2023-12-06T08:01:45.782676+0000","last_clean":"2023-12-06T08:01:45.782676+0000","last_became_active":"2023-12-06T07:57:31.128456+0000","last_became_peered":"2023-12-06T07:57:31.128456+0000","last_unstale":"2023-12-06T08:01:45.782676+0000","last_undegraded":"2023-12-06T08:01:45.782676+0000","last_fullsized":"2023-12-06T08:01:45.782676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:25:07.636439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":162,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-06T08:01:50.331324+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T08:01:50.331324+0000","last_peered":"2023-12-06T08:01:50.331324+0000","last_clean":"2023-12-06T08:01:50.331324+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T08:01:50.331324+0000","last_undegraded":"2023-12-06T08:01:50.331324+0000","last_fullsized":"2023-12-06T08:01:50.331324+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476793,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":244566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":9.0860000000000003,"15min":9.4220000000000006},"min":{"1min":4.1100000000000003,"5min":4.1100000000000003,"15min":4.1100000000000003},"max":{"1min":16.015000000000001,"5min":27.638999999999999,"15min":27.638999999999999},"last":5.5549999999999997},{"interface":"front","average":{"1min":8.9239999999999995,"5min":8.8369999999999997,"15min":8.6850000000000005},"min":{"1min":5.008,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":13.74,"5min":24.085000000000001,"15min":24.085000000000001},"last":10.182}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.744999999999999,"5min":11.625,"15min":12.619},"min":{"1min":6.6299999999999999,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":28.713999999999999,"5min":29.213000000000001,"15min":29.213000000000001},"last":8.266},{"interface":"front","average":{"1min":11.446999999999999,"5min":13.115,"15min":14.191000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":30.204000000000001,"5min":39.337000000000003,"15min":39.337000000000003},"last":9.3650000000000002}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.166,"5min":12.4,"15min":12.557},"min":{"1min":7.3070000000000004,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":27.027000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":10.81},{"interface":"front","average":{"1min":12.935,"5min":13.534000000000001,"15min":13.991},"min":{"1min":9.1110000000000007,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":21.375,"5min":26.388000000000002,"15min":26.388000000000002},"last":11.625999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.247,"5min":14.007999999999999,"15min":13.446},"min":{"1min":8.3350000000000009,"5min":7.8609999999999998,"15min":7.8609999999999998},"max":{"1min":18.373999999999999,"5min":27.263000000000002,"15min":27.263000000000002},"last":15.51},{"interface":"front","average":{"1min":14.763999999999999,"5min":15.885999999999999,"15min":16.044},"min":{"1min":10.791,"5min":7.806,"15min":7.806},"max":{"1min":25.434000000000001,"5min":54.225999999999999,"15min":54.225999999999999},"last":14.144}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.474,"5min":16.591999999999999,"15min":17.013000000000002},"min":{"1min":7.742,"5min":7.742,"15min":7.742},"max":{"1min":23.84,"5min":50.066000000000003,"15min":50.066000000000003},"last":13.118},{"interface":"front","average":{"1min":16.983000000000001,"5min":17.257000000000001,"15min":17.350000000000001},"min":{"1min":11.358000000000001,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":25.030999999999999,"5min":50.478000000000002,"15min":50.478000000000002},"last":18.495999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":16.305,"5min":14.528,"15min":14.744},"min":{"1min":9.1760000000000002,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":23.544,"5min":27.890000000000001,"15min":27.890000000000001},"last":16.256},{"interface":"front","average":{"1min":17.661999999999999,"5min":18.466000000000001,"15min":18.699000000000002},"min":{"1min":13,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":24.808,"5min":44.255000000000003,"15min":44.255000000000003},"last":20.594999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.577999999999999,"5min":17.204999999999998,"15min":18.709},"min":{"1min":8.1319999999999997,"5min":8.0329999999999995,"15min":8.0329999999999995},"max":{"1min":17.728000000000002,"5min":40.832000000000001,"15min":40.832000000000001},"last":19.744},{"interface":"front","average":{"1min":16.411999999999999,"5min":17.524000000000001,"15min":18.338999999999999},"min":{"1min":9.9190000000000005,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":25.026,"5min":39.331000000000003,"15min":39.331000000000003},"last":16.824000000000002}]}]},{"osd":2,"up_from":16,"seq":68719476795,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":249033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.478999999999999,"5min":9.8840000000000003,"15min":10.292},"min":{"1min":4.548,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":23.946999999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":10.711,"5min":10.263999999999999,"15min":10.494999999999999},"min":{"1min":6.9610000000000003,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":31.690000000000001,"5min":34.981000000000002,"15min":34.981000000000002},"last":15.221}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.083,"5min":16.556000000000001,"15min":20.513999999999999},"min":{"1min":6.4059999999999997,"5min":6.4059999999999997,"15min":6.4059999999999997},"max":{"1min":18.251000000000001,"5min":165.68600000000001,"15min":165.68600000000001},"last":9.8290000000000006},{"interface":"front","average":{"1min":13.439,"5min":16.741,"15min":21.100999999999999},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":19.774000000000001,"5min":153.68199999999999,"15min":153.68199999999999},"last":27.974}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.566000000000001,"5min":10.433999999999999,"15min":10.992000000000001},"min":{"1min":6.5579999999999998,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":26.887,"5min":35.935000000000002,"15min":35.935000000000002},"last":12.561},{"interface":"front","average":{"1min":13.198,"5min":15.188000000000001,"15min":15.536},"min":{"1min":8.0129999999999999,"5min":8.0129999999999999,"15min":8.0129999999999999},"max":{"1min":25.497,"5min":30.433,"15min":30.433},"last":18.329000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.345000000000001,"5min":13.98,"15min":16.010000000000002},"min":{"1min":7.1070000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":28.527999999999999,"5min":50.137999999999998,"15min":50.137999999999998},"last":9.1259999999999994},{"interface":"front","average":{"1min":16.071999999999999,"5min":16.396999999999998,"15min":16.539000000000001},"min":{"1min":10.430999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":41.265000000000001,"5min":46.325000000000003,"15min":46.325000000000003},"last":16.393000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.713000000000001,"5min":19.062999999999999,"15min":20.108000000000001},"min":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":42.633000000000003,"5min":50.320999999999998,"15min":50.320999999999998},"last":13.720000000000001},{"interface":"front","average":{"1min":15.576000000000001,"5min":15.792999999999999,"15min":17.120999999999999},"min":{"1min":5.9619999999999997,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":29.931000000000001,"5min":44.323,"15min":44.323},"last":10.427}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":15.397,"5min":17.364999999999998,"15min":19.513000000000002},"min":{"1min":9.6890000000000001,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":44.640999999999998,"5min":104.834,"15min":104.834},"last":24.991},{"interface":"front","average":{"1min":17.795999999999999,"5min":20.042000000000002,"15min":20.542000000000002},"min":{"1min":11.724,"5min":11.308999999999999,"15min":11.308999999999999},"max":{"1min":43.287999999999997,"5min":49.146999999999998,"15min":49.146999999999998},"last":13.16}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.751999999999999,"5min":18.164000000000001,"15min":19.350000000000001},"min":{"1min":8.6020000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":26.753,"5min":42.915999999999997,"15min":42.915999999999997},"last":17.439},{"interface":"front","average":{"1min":16.710999999999999,"5min":17.780000000000001,"15min":18.882000000000001},"min":{"1min":10.266999999999999,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":43.697000000000003,"5min":43.697000000000003,"15min":43.697000000000003},"last":25.602}]}]},{"osd":1,"up_from":16,"seq":68719476795,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":646873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":9.1620000000000008,"5min":10.975,"15min":12.778},"min":{"1min":6.5380000000000003,"5min":4.835,"15min":4.835},"max":{"1min":15.888,"5min":80.640000000000001,"15min":80.640000000000001},"last":9.1379999999999999},{"interface":"front","average":{"1min":12.728,"5min":11.108000000000001,"15min":11.477},"min":{"1min":8.6950000000000003,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":20.853000000000002,"5min":25.390000000000001,"15min":25.390000000000001},"last":12.872999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":16.413,"15min":19.614999999999998},"min":{"1min":8.5640000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":19.486999999999998,"5min":100.703,"15min":100.703},"last":11.109},{"interface":"front","average":{"1min":9.4510000000000005,"5min":14.611000000000001,"15min":17.943000000000001},"min":{"1min":5.4020000000000001,"5min":4.867,"15min":4.867},"max":{"1min":17.934999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":18.405000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":16.853000000000002,"15min":18.071000000000002},"min":{"1min":10.534000000000001,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":31.481999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":12.259},{"interface":"front","average":{"1min":12.157,"5min":11.869,"15min":13.27},"min":{"1min":5.9550000000000001,"5min":5.9550000000000001,"15min":5.9550000000000001},"max":{"1min":30.033000000000001,"5min":46.613,"15min":46.613},"last":7.859}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.465,"5min":15.667999999999999,"15min":17.754999999999999},"min":{"1min":6.6020000000000003,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":27.597000000000001,"5min":81.838999999999999,"15min":81.838999999999999},"last":20.731999999999999},{"interface":"front","average":{"1min":17.969000000000001,"5min":18.931999999999999,"15min":20.847000000000001},"min":{"1min":12.384,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":25.138000000000002,"5min":66.515000000000001,"15min":66.515000000000001},"last":22.526}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.042,"5min":14.403,"15min":15.888999999999999},"min":{"1min":6.4020000000000001,"5min":6.351,"15min":6.351},"max":{"1min":16.806999999999999,"5min":56.235999999999997,"15min":56.235999999999997},"last":21.661000000000001},{"interface":"front","average":{"1min":12.945,"5min":17.241,"15min":19.077000000000002},"min":{"1min":7.1680000000000001,"5min":7.1680000000000001,"15min":7.1680000000000001},"max":{"1min":31.472999999999999,"5min":82.381,"15min":82.381},"last":19.513999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":14.901,"5min":16.239000000000001,"15min":18.032},"min":{"1min":8.5329999999999995,"5min":8.5329999999999995,"15min":8.5329999999999995},"max":{"1min":31.954999999999998,"5min":58.619,"15min":58.619},"last":26.602},{"interface":"front","average":{"1min":15.853,"5min":18.738,"15min":20.597000000000001},"min":{"1min":8.6359999999999992,"5min":7.8529999999999998,"15min":7.8529999999999998},"max":{"1min":33.671999999999997,"5min":83.632000000000005,"15min":83.632000000000005},"last":10.009}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":18.228999999999999,"5min":20.475999999999999,"15min":22.181999999999999},"min":{"1min":13.673,"5min":13.016,"15min":13.016},"max":{"1min":27.407,"5min":62.006999999999998,"15min":62.006999999999998},"last":27.443000000000001},{"interface":"front","average":{"1min":14.981,"5min":17.922000000000001,"15min":19.972000000000001},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":22.795000000000002,"5min":60.698999999999998,"15min":60.698999999999998},"last":32.512999999999998}]}]},{"osd":0,"up_from":16,"seq":68719476796,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":249033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":10.026999999999999,"5min":14.167,"15min":18.539000000000001},"min":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":5.0049999999999999},"max":{"1min":24.733000000000001,"5min":252.88399999999999,"15min":252.88399999999999},"last":9.4499999999999993},{"interface":"front","average":{"1min":10.260999999999999,"5min":14.552,"15min":19.442},"min":{"1min":5.9409999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":17.584,"5min":274.91500000000002,"15min":274.91500000000002},"last":13.497}]},{"osd":2,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":13.33,"5min":14.468,"15min":16.663},"min":{"1min":6.3319999999999999,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":35.383000000000003,"5min":82.866,"15min":82.866},"last":15.997},{"interface":"front","average":{"1min":14.218,"5min":15.249000000000001,"15min":16.135999999999999},"min":{"1min":6.6980000000000004,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":34.289999999999999,"5min":85.991,"15min":85.991},"last":8.609}]},{"osd":3,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":11.598000000000001,"5min":12.791,"15min":14.571},"min":{"1min":7.1109999999999998,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":41.045000000000002,"5min":76.593000000000004,"15min":76.593000000000004},"last":6.3330000000000002},{"interface":"front","average":{"1min":16.559999999999999,"5min":17.657,"15min":19.928000000000001},"min":{"1min":8.875,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":42.284999999999997,"5min":85.162000000000006,"15min":85.162000000000006},"last":15.215999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":14.983000000000001,"5min":16.558,"15min":17.238},"min":{"1min":6.6230000000000002,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":31.988,"5min":76.387,"15min":76.387},"last":11.266999999999999},{"interface":"front","average":{"1min":12.487,"5min":12.813000000000001,"15min":13.67},"min":{"1min":5.8390000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":20.733000000000001,"5min":70.085999999999999,"15min":70.085999999999999},"last":8.9169999999999998}]},{"osd":5,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":15.843999999999999,"5min":21.594000000000001,"15min":26.113},"min":{"1min":10.775,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":27.036000000000001,"5min":144.60499999999999,"15min":144.60499999999999},"last":16.907},{"interface":"front","average":{"1min":16.539999999999999,"5min":21.242999999999999,"15min":24.882999999999999},"min":{"1min":11.016,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":38.125,"5min":142.25,"15min":142.25},"last":10.061}]},{"osd":6,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":17.094999999999999,"5min":16.539000000000001,"15min":17.803000000000001},"min":{"1min":9.2729999999999997,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.847999999999999,"5min":75.471999999999994,"15min":75.471999999999994},"last":10.657999999999999},{"interface":"front","average":{"1min":15.427,"5min":19.684000000000001,"15min":22.881},"min":{"1min":7.8760000000000003,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":29.997,"5min":84.617000000000004,"15min":84.617000000000004},"last":17.675999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":18.831,"5min":20.341000000000001,"15min":21.600999999999999},"min":{"1min":10.420999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":41.003,"5min":84.031000000000006,"15min":84.031000000000006},"last":11.944000000000001},{"interface":"front","average":{"1min":15.169,"5min":16.818999999999999,"15min":18.481000000000002},"min":{"1min":8.3629999999999995,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":27.341999999999999,"5min":77.974000000000004,"15min":77.974000000000004},"last":14.135}]}]},{"osd":4,"up_from":15,"seq":64424509499,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":244566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.013,"5min":8.7430000000000003,"15min":9.3930000000000007},"min":{"1min":4.6109999999999998,"5min":4.234,"15min":4.234},"max":{"1min":48.454000000000001,"5min":48.454000000000001,"15min":48.454000000000001},"last":5.3499999999999996},{"interface":"front","average":{"1min":11.859999999999999,"5min":10.225,"15min":10.269},"min":{"1min":5.3840000000000003,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":44.947000000000003,"5min":44.947000000000003,"15min":44.947000000000003},"last":6.1980000000000004}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.641999999999999,"5min":10.75,"15min":11.603999999999999},"min":{"1min":5.8879999999999999,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":34.880000000000003,"5min":35.524000000000001,"15min":35.524000000000001},"last":9.0109999999999992},{"interface":"front","average":{"1min":11.723000000000001,"5min":12.821,"15min":13.641999999999999},"min":{"1min":7.1970000000000001,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":26.003,"5min":31.027000000000001,"15min":31.027000000000001},"last":8.5690000000000008}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.923999999999999,"5min":12.789999999999999,"15min":12.273999999999999},"min":{"1min":7.1379999999999999,"5min":7.1379999999999999,"15min":7.1379999999999999},"max":{"1min":24.038,"5min":27.353000000000002,"15min":27.353000000000002},"last":12.015000000000001},{"interface":"front","average":{"1min":14.106,"5min":14.686999999999999,"15min":14.646000000000001},"min":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":24.634,"5min":32.915999999999997,"15min":32.915999999999997},"last":9.7620000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.82,"5min":12.065,"15min":11.678000000000001},"min":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"max":{"1min":28.991,"5min":28.991,"15min":28.991},"last":14.086},{"interface":"front","average":{"1min":15.269,"5min":14.705,"15min":14.278},"min":{"1min":10.734,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":35.540999999999997,"5min":35.540999999999997,"15min":35.540999999999997},"last":13.587}]},{"osd":5,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.009,"15min":17.068999999999999},"min":{"1min":10.19,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":27.757000000000001,"5min":53.899999999999999,"15min":53.899999999999999},"last":30.196000000000002},{"interface":"front","average":{"1min":14.430999999999999,"5min":17.372,"15min":17.57},"min":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"max":{"1min":19.631,"5min":55.914000000000001,"15min":55.914000000000001},"last":31.125}]},{"osd":6,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.116,"5min":18.611999999999998,"15min":21.172999999999998},"min":{"1min":9.423,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":20.387,"5min":93.073999999999998,"15min":93.073999999999998},"last":11.590999999999999},{"interface":"front","average":{"1min":16.431000000000001,"5min":15.773,"15min":17.733000000000001},"min":{"1min":6.7779999999999996,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":35.869999999999997,"5min":91.075999999999993,"15min":91.075999999999993},"last":16.102}]},{"osd":7,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":17.501999999999999,"5min":16.015999999999998,"15min":16.25},"min":{"1min":10.579000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":31.844000000000001,"5min":33.988999999999997,"15min":33.988999999999997},"last":14.704000000000001},{"interface":"front","average":{"1min":17.699999999999999,"5min":17.460999999999999,"15min":16.974},"min":{"1min":11.65,"5min":10.885,"15min":10.885},"max":{"1min":37.154000000000003,"5min":37.154000000000003,"15min":37.154000000000003},"last":16.899000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542201,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":244566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.220000000000001,"5min":9.7940000000000005,"15min":10.138},"min":{"1min":4.718,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":35.454000000000001,"5min":70.965000000000003,"15min":70.965000000000003},"last":5.6020000000000003},{"interface":"front","average":{"1min":8.1180000000000003,"5min":10.308,"15min":11.661},"min":{"1min":5.3979999999999997,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":17.193999999999999,"5min":73.582999999999998,"15min":73.582999999999998},"last":7.7320000000000002}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":12.234999999999999,"5min":15.962999999999999,"15min":18.359999999999999},"min":{"1min":7.9580000000000002,"5min":4.383,"15min":4.383},"max":{"1min":22.164000000000001,"5min":217.595,"15min":217.595},"last":14.672000000000001},{"interface":"front","average":{"1min":13.619999999999999,"5min":17.018999999999998,"15min":21.198},"min":{"1min":7.0730000000000004,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":37.552,"5min":284.96499999999997,"15min":284.96499999999997},"last":9.032}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.404999999999999,"5min":13.512,"15min":13.757999999999999},"min":{"1min":6.7140000000000004,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.434999999999999,"5min":46.024999999999999,"15min":46.024999999999999},"last":10.657999999999999},{"interface":"front","average":{"1min":15.532,"5min":15.390000000000001,"15min":16.452000000000002},"min":{"1min":8.4120000000000008,"5min":7.7610000000000001,"15min":7.7610000000000001},"max":{"1min":40.298999999999999,"5min":46.966999999999999,"15min":46.966999999999999},"last":12.704000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":13.816000000000001,"5min":15.863,"15min":15.851000000000001},"min":{"1min":9.4450000000000003,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":21.167000000000002,"5min":65.719999999999999,"15min":65.719999999999999},"last":11.871},{"interface":"front","average":{"1min":18.998000000000001,"5min":19.789000000000001,"15min":22.832999999999998},"min":{"1min":11.177,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":41.137,"5min":189.84999999999999,"15min":189.84999999999999},"last":15.611000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.972999999999999,"5min":17.817,"15min":16.527999999999999},"min":{"1min":9.4909999999999997,"5min":9.3409999999999993,"15min":9.3409999999999993},"max":{"1min":42.905999999999999,"5min":66.349999999999994,"15min":66.349999999999994},"last":13.678000000000001},{"interface":"front","average":{"1min":16.096,"5min":15.782999999999999,"15min":15.013999999999999},"min":{"1min":11.175000000000001,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":25.131,"5min":39.520000000000003,"15min":39.520000000000003},"last":11.276}]},{"osd":5,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.574000000000002,"5min":18.863,"15min":18.902999999999999},"min":{"1min":9.7530000000000001,"5min":9.7530000000000001,"15min":9.7530000000000001},"max":{"1min":41.997,"5min":50.945,"15min":50.945},"last":18.010999999999999},{"interface":"front","average":{"1min":16.722999999999999,"5min":14.596,"15min":14.446999999999999},"min":{"1min":7.1550000000000002,"5min":6.4960000000000004,"15min":6.4960000000000004},"max":{"1min":38.826000000000001,"5min":51.948,"15min":51.948},"last":6.4690000000000003}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.364000000000001,"5min":17.041,"15min":16.187000000000001},"min":{"1min":12.323,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":52.875,"5min":52.875,"15min":52.875},"last":9.9730000000000008},{"interface":"front","average":{"1min":22.992999999999999,"5min":20.244,"15min":19.75},"min":{"1min":14.509,"5min":11.18,"15min":11.18},"max":{"1min":51.877000000000002,"5min":71.346999999999994,"15min":71.346999999999994},"last":19.332000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1064960,"data_stored":642406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":9.4220000000000006,"5min":9.2490000000000006,"15min":9.1739999999999995},"min":{"1min":5.2039999999999997,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":24.896000000000001,"5min":26.117999999999999,"15min":26.117999999999999},"last":9.5269999999999992},{"interface":"front","average":{"1min":9.7729999999999997,"5min":9.5340000000000007,"15min":9.5999999999999996},"min":{"1min":5.5110000000000001,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":23.355,"5min":28.145,"15min":28.145},"last":8.5139999999999993}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":10.904999999999999,"5min":12.387,"15min":13.345000000000001},"min":{"1min":5.2389999999999999,"5min":4.8659999999999997,"15min":4.8659999999999997},"max":{"1min":20.106000000000002,"5min":67.347999999999999,"15min":67.347999999999999},"last":21.478000000000002},{"interface":"front","average":{"1min":9.968,"5min":12.228,"15min":12.582000000000001},"min":{"1min":5.343,"5min":5.343,"15min":5.343},"max":{"1min":19.221,"5min":39.052,"15min":39.052},"last":17.82}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.557,"5min":12.074999999999999,"15min":12.106999999999999},"min":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"max":{"1min":24.247,"5min":31.873000000000001,"15min":31.873000000000001},"last":16.395},{"interface":"front","average":{"1min":14.055,"5min":14.414,"15min":15.505000000000001},"min":{"1min":7.6829999999999998,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":29.513999999999999,"5min":34.585000000000001,"15min":34.585000000000001},"last":24.742999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":14.154999999999999,"5min":14.811,"15min":15.292999999999999},"min":{"1min":8.4160000000000004,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":28.643000000000001,"5min":30.355,"15min":30.355},"last":19.616},{"interface":"front","average":{"1min":15.125,"5min":14.696999999999999,"15min":15.029999999999999},"min":{"1min":9.0920000000000005,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":30.988,"5min":31.792000000000002,"15min":31.792000000000002},"last":20.856999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.83,"5min":20.387,"15min":26.422999999999998},"min":{"1min":10.773999999999999,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":24.518000000000001,"5min":334.86500000000001,"15min":334.86500000000001},"last":46.381},{"interface":"front","average":{"1min":13.234999999999999,"5min":20.062000000000001,"15min":26.206},"min":{"1min":7.5810000000000004,"5min":7.431,"15min":7.431},"max":{"1min":20.047999999999998,"5min":331.57900000000001,"15min":331.57900000000001},"last":45.612000000000002}]},{"osd":6,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.187999999999999,"5min":18.253,"15min":19.466999999999999},"min":{"1min":10.048,"5min":9.1310000000000002,"15min":9.1310000000000002},"max":{"1min":35.459000000000003,"5min":46.034999999999997,"15min":46.034999999999997},"last":25.373999999999999},{"interface":"front","average":{"1min":15.515000000000001,"5min":17.102,"15min":18.291},"min":{"1min":8.343,"5min":8.343,"15min":8.343},"max":{"1min":34.116,"5min":35.680999999999997,"15min":35.680999999999997},"last":23.457999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.087,"5min":21.350000000000001,"15min":28},"min":{"1min":11.781000000000001,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":19.779,"5min":332.322,"15min":332.322},"last":17.111999999999998},{"interface":"front","average":{"1min":19.039999999999999,"5min":24.308,"15min":31.135999999999999},"min":{"1min":13.331,"5min":10.467000000000001,"15min":10.467000000000001},"max":{"1min":26.495999999999999,"5min":325.93599999999998,"15min":325.93599999999998},"last":29.148}]}]},{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":249033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.3860000000000001,"5min":8.7629999999999999,"15min":9.2940000000000005},"min":{"1min":4.7590000000000003,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.015000000000001,"5min":20.558,"15min":20.558},"last":14.029999999999999},{"interface":"front","average":{"1min":11.315,"5min":11.834,"15min":11.933999999999999},"min":{"1min":6.5359999999999996,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":17.52,"5min":20.893999999999998,"15min":20.893999999999998},"last":15.173}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.4,"5min":14.412000000000001,"15min":16.027999999999999},"min":{"1min":7.2130000000000001,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":18.568999999999999,"5min":51.890000000000001,"15min":51.890000000000001},"last":24.308},{"interface":"front","average":{"1min":7.6219999999999999,"5min":8.7490000000000006,"15min":8.5109999999999992},"min":{"1min":5.6360000000000001,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":14.318,"5min":37.042999999999999,"15min":37.042999999999999},"last":17.858000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.321999999999999,"5min":12.026,"15min":12.414},"min":{"1min":4.359,"5min":4.359,"15min":4.359},"max":{"1min":20.170999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":25.84},{"interface":"front","average":{"1min":11.811999999999999,"5min":11.840999999999999,"15min":11.384},"min":{"1min":7.7549999999999999,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":18.934999999999999,"5min":35.002000000000002,"15min":35.002000000000002},"last":13.028}]},{"osd":3,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.914999999999999,"5min":19.931999999999999,"15min":26.753},"min":{"1min":7.9039999999999999,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":15.723000000000001,"5min":228.17099999999999,"15min":228.17099999999999},"last":27.279},{"interface":"front","average":{"1min":14.678000000000001,"5min":22.178999999999998,"15min":29.501000000000001},"min":{"1min":9.9580000000000002,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":18.475000000000001,"5min":270.13,"15min":270.13},"last":35.183999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.047000000000001,"5min":14.993,"15min":16.600999999999999},"min":{"1min":10.432,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":16.914999999999999,"5min":86.980999999999995,"15min":86.980999999999995},"last":18.478999999999999},{"interface":"front","average":{"1min":14.48,"5min":17.702000000000002,"15min":19.783999999999999},"min":{"1min":9.2579999999999991,"5min":7.8719999999999999,"15min":7.8719999999999999},"max":{"1min":18.155000000000001,"5min":99.021000000000001,"15min":99.021000000000001},"last":28.888999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.275,"5min":19.556999999999999,"15min":21.327000000000002},"min":{"1min":11.198,"5min":10.254,"15min":10.254},"max":{"1min":19.477,"5min":97.191000000000003,"15min":97.191000000000003},"last":41.997999999999998},{"interface":"front","average":{"1min":16.013000000000002,"5min":18.779,"15min":20.585999999999999},"min":{"1min":10.664,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":19.811,"5min":64.048000000000002,"15min":64.048000000000002},"last":19.091999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":14.233000000000001,"5min":16.07,"15min":17.100000000000001},"min":{"1min":9.3729999999999993,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":18.184000000000001,"5min":74.064999999999998,"15min":74.064999999999998},"last":15.983000000000001},{"interface":"front","average":{"1min":16.649000000000001,"5min":20.157,"15min":21.806000000000001},"min":{"1min":10.581,"5min":10.055,"15min":10.055},"max":{"1min":20.405000000000001,"5min":92.477999999999994,"15min":92.477999999999994},"last":42.814}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:01:56.176 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [0, 3] 2023-12-06T08:01:56.177 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.3', '0', '3'] 2023-12-06T08:01:56.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.3 0 3 2023-12-06T08:01:56.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:56.202 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:56.245 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:56.245+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:56.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:01:56.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:56.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:56.326 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:56.326+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:56.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2023-12-06T08:01:56.392 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:56.392 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:56.395 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:56.395 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:56.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:01:56.440 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:56.440+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:56.507 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:56.508 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:56.528 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:56.529+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:56.553 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:56.553 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:56.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:01:56.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:56.629 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:56.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:56.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:56.694 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:56.735 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:56.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:01:56.735 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:01:56.735 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:01:56.735 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:56.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:01:56.748 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:56.747+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:56.831 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:56.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 256.29855938899999, 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:56.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:01:56.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:56.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:56.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 76.181382042999999, 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:56.883 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:56.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:56.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.888 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.174577827, 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:01:56.890 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:56.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:01:56.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:01:56.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:01:56.894 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:01:56.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:01:56.940 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:56.940+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:57.029 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.029 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:57.029 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:57.029 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:57.032 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:57.032+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:57.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:01:57.134 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:57.134+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:57.209 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.209 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:57.209 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:57.209 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:57.209 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:57.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:01:57.261 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:57.261+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:57.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2023-12-06T08:01:57.361 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:57.361+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:57.396 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.397 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:57.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:57.397 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:57.397 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:57.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:01:57.438 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:57.438+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:57.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:01:57.539 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:57.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:57.585 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.585 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:57.585 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:57.585 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:57.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:01:57.640 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:57.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:57.724 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:57.724 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:57.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:01:57.777 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:57.777+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:57.841 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:57.841+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 277.79533326900003, 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.32421981599998, 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:01:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:01:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:01:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.09402743700002, 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:01:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:01:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.093319888, 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.09268289400001, 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.035617699, 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.03412692799998, 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:01:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 276.00177726599998, 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:01:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 275.99514649100001, 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:01:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 275.99449580999999, 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:01:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:01:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.99102311899998, 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:01:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:01:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.98988439700003, 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:01:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:01:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:01:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.98893670000001, 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.96057270599999, 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:01:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:01:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:01:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.932363189, 2023-12-06T08:01:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:01:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.32216525500002, 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:01:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.32152041500001, 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:01:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:01:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.30105366100003, 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:01:58.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:01:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:01:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:01:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.30030474500001, 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:01:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:01:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:01:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.29966267600003, 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:01:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:01:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:58.033 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:58.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:01:58.035 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:58.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:58.130 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.3 pg_upmap mapping to [0,3] 2023-12-06T08:01:58.135 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:58.135+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:58.212 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:58.212 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:58.212 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:58.212 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:58.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:01:58.266 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:58.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:58.340 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:58.341+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:58.352 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:58.352 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:58.352 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:58.352 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:58.352 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:58.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:01:58.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2023-12-06T08:01:58.444 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:58.444+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:58.545 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:58.545+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:58.647 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:58.647+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:58.679 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:58.679 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:58.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:58.679 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:01:58.680 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:58.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:01:58.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:01:58.773 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:58.773+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:58.851 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:58.851+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:58.878 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:58.878 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:01:58.878 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:58.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:01:58.975 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:58.974+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:59.000 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:01:59.000 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:01:59.055 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:59.054+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:59.060 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:59.061 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:01:59.061 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:01:59.061 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:01:59.061 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:59.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:01:59.089 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:01:59.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:01:59.112 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:01:59.156 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:01:59.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:01:59.220 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:01:59.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:01:59.258 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 259.83790639599999, 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:59.259 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:59.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:59.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:59.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 79.674407630999994, 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:59.263 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:59.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:01:59.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.667346617, 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:01:59.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:01:59.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:01:59.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:01:59.270 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:01:59.271 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:01:59.272 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:01:59.259+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:01:59.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:01:59.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:01:59.327 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:01:59.360 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:01:59.360+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:01:59.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:01:59.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:01:59.454 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:59.455 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:59.462 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:01:59.462+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:01:59.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:01:59.519 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:01:59.520 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:01:59.563 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:01:59.564+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:01:59.617 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:59.617 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:59.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:01:59.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:01:59.627 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:01:59.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2023-12-06T08:01:59.667 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:01:59.668+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:01:59.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:01:59.718 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:01:59.772 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:01:59.772+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:01:59.816 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:01:59.816 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:01:59.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:01:59.872 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:01:59.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:01:59.959 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.079 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:00.081 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:00.080+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:00.182 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:00.182+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:00.250 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.251 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:00.283 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:00.283+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:00.384 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:00.384+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:00.431 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.431 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:00.485 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:00.485+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:00.571 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.571 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:00.622 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:00.623+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:00.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2023-12-06T08:02:00.686 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:00.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:00.751 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.751 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:00.802 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:00.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:00.888 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:00.888+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:00.931 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:00.931 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:00.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:00.988 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:00.989+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:01.067 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:01.068 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:01.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:01.089 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:01.089+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:01.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:02:01.190 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:01.191+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:01.247 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:01.247 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:01.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:01.297 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:01.298+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:01.392 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:01.392+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:01.423 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:01.423 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:01.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:01.493 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:01.493+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:01.581 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:01.581 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:01.596 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:01.595+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:01.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:01.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2023-12-06T08:02:01.695 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:01.695+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:01.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:01.790 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:01.790 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:01.795 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:01.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:01.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:01.896 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:01.896+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:01.997 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:01.997+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:02.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:02.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:02.004 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:02.005 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:02.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:02.005 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:02.005 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:02.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:02.098 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:02.098+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:02.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:02.114 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:02.188 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:02.188 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:02.188 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:02.188 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:02.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:02.217 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:02.218 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:02.218 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:02.218+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:02.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:02.327 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:02.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:02.361 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:02.361 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:02.361 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:02.361 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:02.361 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:02.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:02.402 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:02.402+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:02.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:02.429 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:02.503 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:02.503+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:02.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:02.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:02.610 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:02.610+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:02.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:02.610 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:02.614 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 282.46685099299998, 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:02:02.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:02:02.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:02:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:02:02.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.99573753999999, 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.76554516099998, 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.76483761200001, 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:02:02.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.76420061800002, 2023-12-06T08:02:02.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.70713542300001, 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:02:02.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:02:02.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:02:02.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:02:02.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.70564465199999, 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.653 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:02:02.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:02:02.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.67329498999999, 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:02:02.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:02:02.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:02.665 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.66666421500003, 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:02:02.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:02:02.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:02.673 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:02.673 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.666013534, 2023-12-06T08:02:02.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:02:02.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:02:02.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.678 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 279.66254084299999, 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.684 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 279.66140212099998, 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:02:02.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:02:02.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:02:02.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 279.66045442400002, 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.692 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:02:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 279.63209043000001, 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.697 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:02:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 279.60388091300001, 2023-12-06T08:02:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:02:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:02:02.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:02:02.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.707 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 273.99368297900003, 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 273.99303813900002, 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:02:02.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:02:02.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 273.97257138499998, 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.718 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:02:02.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:02:02.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:02:02.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:02:02.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 273.97182246900002, 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 273.97118039999998, 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:02.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:02:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:02:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:02.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:02.736 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:02.737 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:02.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:02.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:02.739 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:02.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2023-12-06T08:02:02.740 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:02.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:02.787 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:02:02.845 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:02.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:02.913 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:02.914 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:02.914 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:02.914 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:02.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:02.941 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:02.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:03.042 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:03.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:03.090 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:03.090 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:03.091 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:03.091 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:03.091 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:03.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:03.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:03.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-06T08:02:03.148 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:03.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:03.249 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:03.249+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:03.305 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:03.305 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:03.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:03.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:03.306 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:03.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:03.362 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:03.362+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:03.447 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:03.447 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:03.447 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:03.447 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:03.450 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:03.451+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:03.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:03.552 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:03.552+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:03.633 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:03.633 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:03.634 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:03.634 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:03.634 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:03.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:03.654 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:03.654+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:03.755 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:03.755+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:03.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2023-12-06T08:02:03.831 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:03.831 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 264.41025481999998, 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.832 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:03.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 84.246756055000006, 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:03.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:03.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:03.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:03.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.239695041000001, 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:03.841 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:03.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:03.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:03.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:03.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:03.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:03.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:03.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:03.846 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:03.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:03.889 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:03.889+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:03.958 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:03.957+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:03.984 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:03.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:03.985 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:03.985 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:03.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:04.058 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:04.059+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:04.123 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:04.123 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:04.123 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:04.123 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:04.123 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:04.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:04.159 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:04.159+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:04.260 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:04.261+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:04.300 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:04.300 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:04.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:04.300 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:04.301 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:04.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:04.393 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:04.394+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:04.462 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:04.462+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:04.468 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2023-12-06T08:02:04.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-06T08:02:04.483 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:04.483 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:04.484 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:04.484 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:04.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:04.579 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:04.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:04.664 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:04.665+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:04.667 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:04.667 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:04.667 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:04.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:04.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:04.765 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:04.765+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:04.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:02:04.866 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:04.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 283.90364360799998, 2023-12-06T08:02:04.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:02:04.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:02:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:02:04.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 283.23986260700002, 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.889 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:04.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:02:04.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 283.236581978, 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:02:04.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:02:04.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.903 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.93180536199998, 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.907 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.93096226500001, 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:02:04.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:02:04.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:02:04.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.90842675900001, 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:02:04.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.90643433999998, 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:02:04.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:02:04.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.90590207499997, 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.88897716600002, 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:04.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:02:04.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.928 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.88814053200002, 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:04.932 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.86369326400001, 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.8577909, 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:02:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:02:04.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:02:04.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:02:04.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.85308490400001, 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.940 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.85078302900001, 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:02:04.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.85028470600002, 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.948 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:02:04.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:02:04.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:02:04.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:02:04.952 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.849830795, 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:04.956 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 281.84860839499999, 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:02:04.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 276.20344241399999, 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.960 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:02:04.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 276.19600399900003, 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 276.185930627, 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:02:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:04.972 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:04.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:04.974 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:04.973+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:05.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2023-12-06T08:02:05.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:05.040 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:05.080 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:05.080+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:05.142 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:05.142 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:05.142 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:05.142 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:05.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:05.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:05.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:05.179 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:05.179+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:05.252 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:05.252 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:05.277 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:05.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:05.277 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:05.277 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:05.277 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:05.277 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:05.278 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:05.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:05.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:05.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:05.379 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:05.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:05.415 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:05.415 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:05.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:05.415 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:05.415 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:05.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:05.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:05.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:05.479 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:05.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:05.566 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:05.566 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:05.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:05.581 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:05.580+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:05.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:05.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:05.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:05.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:05.681 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:05.681+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:05.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:05.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:05.772 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:05.772 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:05.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:05.783 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:05.783+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:05.884 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:05.884+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:05.910 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:05.910 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:05.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:05.983 INFO:tasks.mon_recovery:adding mon b back in 2023-12-06T08:02:05.983 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-06T08:02:05.983 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-06T08:02:05.985 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:05.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:05.985 INFO:tasks.ceph.mon.b:Started 2023-12-06T08:02:05.986 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T08:02:05.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:06.057 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.057 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:06.110 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:06.110+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:06.186 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:06.186+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:06.216 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.216 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:06.310 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:06.310+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:06.388 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:06.389+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:06.406 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.406 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:06.500 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:06.500+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:06.591 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:06.591+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:06.607 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:02:06.607 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2023-12-06T08:02:06.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-06T08:02:06.635 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.636 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:06.681 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:06.696 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:06.696+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:06.795 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:06.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:06.822 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.823 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:06.896 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:06.896+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:06.966 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:06.966 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:06.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:06.998 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:06.997+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:07.099 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:07.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:07.113 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.113 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:07.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:07.198 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:07.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:07.300 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:07.299+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:07.302 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.303 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:07.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:07.400 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:07.400+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:07.444 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.445 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:07.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:07.539 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:07.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:07.602 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:07.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:07.627 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.627 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:07.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:07.702 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:07.703+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:07.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:07.803 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:07.803+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:07.806 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.806 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:07.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:07.904 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:07.905+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:07.999 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:07.999 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:08.005 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:08.005+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:08.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:08.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:08.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:08.106 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:08.107+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:08.122 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:08.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:08.141 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:08.141 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:08.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:08.161 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2023-12-06T08:02:08.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-06T08:02:08.221 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:08.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:08.222 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:08.221+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:08.309 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:08.309+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:08.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:08.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:08.346 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:08.346 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:08.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:08.410 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:08.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:08.438 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:08.439+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:08.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:08.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:08.523 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:08.523+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:08.537 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:08.538 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:08.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:08.611 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:08.611+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:08.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:08.615 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:08.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:02:08.702 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:08.702 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:08.712 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:08.712+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:08.717 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:08.718 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:08.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:08.813 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:08.814+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:08.897 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:08.898 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:08.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:08.946 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:08.947+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:09.015 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:09.015+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 288.13322496799998, 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:02:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:02:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:02:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.46944396700002, 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.466163338, 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.16138672199997, 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:02:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:02:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:02:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.160543625, 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:02:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:02:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:02:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.13800811900001, 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:02:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.13601569999997, 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:02:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:02:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.13548343500003, 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:02:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.11855852600002, 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.11772189200002, 2023-12-06T08:02:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.093274624, 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.08737226, 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.08266626400001, 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.171 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:09.117+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:02:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.08036438900001, 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.07986606600002, 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.079412155, 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:02:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.07818975499998, 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:02:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:02:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:02:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 280.43302377399999, 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:02:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:02:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:02:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:02:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 280.42558535900002, 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:02:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:02:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 280.415511987, 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:02:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:09.202 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:09.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:09.257 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:09.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:09.257 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2023-12-06T08:02:09.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-12-06T08:02:09.318 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:09.318+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:09.343 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:09.343 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:09.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:02:09.419 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:09.419+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:09.520 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:09.520 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:09.520 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:09.520 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:09.520 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:09.520 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:09.519+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:09.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:09.551 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:05.984 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.553 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:05.986 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.582 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:06.014 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.621 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:09.620+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:09.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:09.697 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:09.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:09.698 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:09.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:09.762 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:09.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:09.823 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:09.823+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:09.878 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:25.789 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.883 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:09.884 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:09.884 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:09.884 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:09.889 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:25.801 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:09.917 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:25.829 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:09.934 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:09.934+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:10.025 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:10.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:10.060 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:10.060 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:10.060 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:10.060 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:10.060 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:10.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:10.126 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:10.125+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:10.205 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 269.62581693700002, 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:10.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 89.508639591000005, 2023-12-06T08:02:10.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:10.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:10.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.501835374999999, 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.213 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:10.216 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:10.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:10.258 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:10.258+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:10.328 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:10.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:10.386 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:10.386 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:10.386 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:10.386 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:10.392 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":35,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:10.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:10.448 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:10.448+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:10.529 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:10.529+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2023-12-06T08:02:10.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-12-06T08:02:10.591 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:10.591 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:10.591 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:10.591 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:10.591 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:10.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:10.631 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:10.630+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:10.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:10.766 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:10.766+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:10.791 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:10.791 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:10.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:10.791 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:10.791 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:10.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:10.832 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:10.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:10.934 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:10.933+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:10.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:10.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:10.994 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:10.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:10.994 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:10.994 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:11.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:11.035 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:11.035+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:11.057 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:11.058 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:11.138 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:11.137+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:11.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:11.152 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:11.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:11.159 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:11.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:11.238 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:11.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:11.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:11.246 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:11.341 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:11.340+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:11.349 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:11.349 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:11.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:11.354 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:11.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:11.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:02:11.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:11.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:11.441 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:11.441+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:11.499 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:11.499 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:11.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:11.538 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:11.538 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:11.565 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:11.564+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:11.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:11.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:11.643 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:11.643+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:11.677 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:11.677 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:11.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:11.749 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:11.748+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:11.846 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:11.845+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:11.869 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:11.869 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:11.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:11.900 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2023-12-06T08:02:11.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-12-06T08:02:11.957 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:11.957+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:12.047 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:12.047+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:12.148 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:12.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:12.183 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.183 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:12.249 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:12.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:12.320 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.321 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:12.374 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:12.374+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:12.462 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.462 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.464 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:12.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:12.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:12.556 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:12.556+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:12.641 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.641 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:12.692 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:12.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:12.754 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:12.754+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:12.826 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.826 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:12.855 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:12.854+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:12.956 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:12.956+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:12.967 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:12.967 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:12.971 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2023-12-06T08:02:12.971 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:02:12.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:12.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-12-06T08:02:13.075 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:13.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.147 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:13.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:13.158 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:13.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:13.275 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:13.274+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.289 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:13.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:13.383 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:13.383+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:13.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:13.461 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:13.460+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:13.469 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.469 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:13.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:13.562 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:13.561+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:13.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:13.666 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.666 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:13.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:13.693 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:13.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:13.764 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:13.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:13.818 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.819 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:13.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:13.865 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:13.864+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:13.906 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:13.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:13.966 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:13.965+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:13.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:13.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:13.998 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:13.999 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:14.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:14.067 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:14.066+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:14.068 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:08.022 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:14.083 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:14.083 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:14.137 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:14.137 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:14.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:14.170 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":39,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:14.193 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:14.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:14.199 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:14.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:14.261 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2023-12-06T08:02:14.271 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:14.270+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:14.275 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-06T08:02:14.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:02:14.288 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:14.288 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:14.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:14.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:14.303 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:14.318 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:08.270 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:14.370 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:14.369+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:14.413 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:14.414 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:14.461 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:14.461 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:14.461 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:14.461 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:14.471 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:14.469+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:14.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:14.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:14.517 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:14.572 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:14.571+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:14.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:14.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:14.656 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:14.656 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:14.656 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:14.656 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:14.656 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:14.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:14.706 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:14.705+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:14.774 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:14.773+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:14.848 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:14.848 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:14.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:14.848 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:14.848 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:14.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:14.874 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:14.874+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:14.953 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:14.954 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:02:14.966 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2023-12-06T08:02:14.194970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223160,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751560,"statfs":{"total":773094113280,"available":772865597440,"internally_reserved":0,"allocated":7143424,"data_stored":3129952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.344274"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.099848+0000","last_change":"2023-12-06T08:01:15.480948+0000","last_active":"2023-12-06T08:02:09.099848+0000","last_peered":"2023-12-06T08:02:09.099848+0000","last_clean":"2023-12-06T08:02:09.099848+0000","last_became_active":"2023-12-06T08:01:15.474602+0000","last_became_peered":"2023-12-06T08:01:15.474602+0000","last_unstale":"2023-12-06T08:02:09.099848+0000","last_undegraded":"2023-12-06T08:02:09.099848+0000","last_fullsized":"2023-12-06T08:02:09.099848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:07:38.010954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.440024+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T08:02:09.440024+0000","last_peered":"2023-12-06T08:02:09.440024+0000","last_clean":"2023-12-06T08:02:09.440024+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T08:02:09.440024+0000","last_undegraded":"2023-12-06T08:02:09.440024+0000","last_fullsized":"2023-12-06T08:02:09.440024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.092542+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:02:09.092542+0000","last_peered":"2023-12-06T08:02:09.092542+0000","last_clean":"2023-12-06T08:02:09.092542+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:02:09.092542+0000","last_undegraded":"2023-12-06T08:02:09.092542+0000","last_fullsized":"2023-12-06T08:02:09.092542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.791210+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T08:02:09.791210+0000","last_peered":"2023-12-06T08:02:09.791210+0000","last_clean":"2023-12-06T08:02:09.791210+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T08:02:09.791210+0000","last_undegraded":"2023-12-06T08:02:09.791210+0000","last_fullsized":"2023-12-06T08:02:09.791210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:08.226461+0000","last_change":"2023-12-06T08:01:04.450104+0000","last_active":"2023-12-06T08:02:08.226461+0000","last_peered":"2023-12-06T08:02:08.226461+0000","last_clean":"2023-12-06T08:02:08.226461+0000","last_became_active":"2023-12-06T08:01:04.441340+0000","last_became_peered":"2023-12-06T08:01:04.441340+0000","last_unstale":"2023-12-06T08:02:08.226461+0000","last_undegraded":"2023-12-06T08:02:08.226461+0000","last_fullsized":"2023-12-06T08:02:08.226461+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:26:50.232248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.436089+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T08:02:09.436089+0000","last_peered":"2023-12-06T08:02:09.436089+0000","last_clean":"2023-12-06T08:02:09.436089+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T08:02:09.436089+0000","last_undegraded":"2023-12-06T08:02:09.436089+0000","last_fullsized":"2023-12-06T08:02:09.436089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-12-06T08:02:12.773593+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:02:12.773593+0000","last_peered":"2023-12-06T08:02:12.773593+0000","last_clean":"2023-12-06T08:02:12.773593+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:02:12.773593+0000","last_undegraded":"2023-12-06T08:02:12.773593+0000","last_fullsized":"2023-12-06T08:02:12.773593+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.200780+0000","last_change":"2023-12-06T08:01:59.482848+0000","last_active":"2023-12-06T08:02:09.200780+0000","last_peered":"2023-12-06T08:02:09.200780+0000","last_clean":"2023-12-06T08:02:09.200780+0000","last_became_active":"2023-12-06T08:01:59.452600+0000","last_became_peered":"2023-12-06T08:01:59.452600+0000","last_unstale":"2023-12-06T08:02:09.200780+0000","last_undegraded":"2023-12-06T08:02:09.200780+0000","last_fullsized":"2023-12-06T08:02:09.200780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:39:53.539482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":171,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.088553+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T08:02:09.088553+0000","last_peered":"2023-12-06T08:02:09.088553+0000","last_clean":"2023-12-06T08:02:09.088553+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T08:02:09.088553+0000","last_undegraded":"2023-12-06T08:02:09.088553+0000","last_fullsized":"2023-12-06T08:02:09.088553+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476797,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":294033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":9.0860000000000003,"15min":9.4220000000000006},"min":{"1min":4.1100000000000003,"5min":4.1100000000000003,"15min":4.1100000000000003},"max":{"1min":16.015000000000001,"5min":27.638999999999999,"15min":27.638999999999999},"last":8.1229999999999993},{"interface":"front","average":{"1min":8.9239999999999995,"5min":8.8369999999999997,"15min":8.6850000000000005},"min":{"1min":5.008,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":13.74,"5min":24.085000000000001,"15min":24.085000000000001},"last":8.6159999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.744999999999999,"5min":11.625,"15min":12.619},"min":{"1min":6.6299999999999999,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":28.713999999999999,"5min":29.213000000000001,"15min":29.213000000000001},"last":18.27},{"interface":"front","average":{"1min":11.446999999999999,"5min":13.115,"15min":14.191000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":30.204000000000001,"5min":39.337000000000003,"15min":39.337000000000003},"last":9.9480000000000004}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.166,"5min":12.4,"15min":12.557},"min":{"1min":7.3070000000000004,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":27.027000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":10.521000000000001},{"interface":"front","average":{"1min":12.935,"5min":13.534000000000001,"15min":13.991},"min":{"1min":9.1110000000000007,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":21.375,"5min":26.388000000000002,"15min":26.388000000000002},"last":11.228}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.247,"5min":14.007999999999999,"15min":13.446},"min":{"1min":8.3350000000000009,"5min":7.8609999999999998,"15min":7.8609999999999998},"max":{"1min":18.373999999999999,"5min":27.263000000000002,"15min":27.263000000000002},"last":12.381},{"interface":"front","average":{"1min":14.763999999999999,"5min":15.885999999999999,"15min":16.044},"min":{"1min":10.791,"5min":7.806,"15min":7.806},"max":{"1min":25.434000000000001,"5min":54.225999999999999,"15min":54.225999999999999},"last":17.497}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.474,"5min":16.591999999999999,"15min":17.013000000000002},"min":{"1min":7.742,"5min":7.742,"15min":7.742},"max":{"1min":23.84,"5min":50.066000000000003,"15min":50.066000000000003},"last":14.228999999999999},{"interface":"front","average":{"1min":16.983000000000001,"5min":17.257000000000001,"15min":17.350000000000001},"min":{"1min":11.358000000000001,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":25.030999999999999,"5min":50.478000000000002,"15min":50.478000000000002},"last":16.597999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":16.305,"5min":14.528,"15min":14.744},"min":{"1min":9.1760000000000002,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":23.544,"5min":27.890000000000001,"15min":27.890000000000001},"last":13.640000000000001},{"interface":"front","average":{"1min":17.661999999999999,"5min":18.466000000000001,"15min":18.699000000000002},"min":{"1min":13,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":24.808,"5min":44.255000000000003,"15min":44.255000000000003},"last":19.170999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.577999999999999,"5min":17.204999999999998,"15min":18.709},"min":{"1min":8.1319999999999997,"5min":8.0329999999999995,"15min":8.0329999999999995},"max":{"1min":17.728000000000002,"5min":40.832000000000001,"15min":40.832000000000001},"last":15.786},{"interface":"front","average":{"1min":16.411999999999999,"5min":17.524000000000001,"15min":18.338999999999999},"min":{"1min":9.9190000000000005,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":25.026,"5min":39.331000000000003,"15min":39.331000000000003},"last":14.85}]}]},{"osd":2,"up_from":16,"seq":68719476798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":280529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.478999999999999,"5min":9.8840000000000003,"15min":10.292},"min":{"1min":4.548,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":23.946999999999999},"last":6.6180000000000003},{"interface":"front","average":{"1min":10.711,"5min":10.263999999999999,"15min":10.494999999999999},"min":{"1min":6.9610000000000003,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":31.690000000000001,"5min":34.981000000000002,"15min":34.981000000000002},"last":10.526}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.083,"5min":16.556000000000001,"15min":20.513999999999999},"min":{"1min":6.4059999999999997,"5min":6.4059999999999997,"15min":6.4059999999999997},"max":{"1min":18.251000000000001,"5min":165.68600000000001,"15min":165.68600000000001},"last":13.292999999999999},{"interface":"front","average":{"1min":13.439,"5min":16.741,"15min":21.100999999999999},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":19.774000000000001,"5min":153.68199999999999,"15min":153.68199999999999},"last":14.071}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.566000000000001,"5min":10.433999999999999,"15min":10.992000000000001},"min":{"1min":6.5579999999999998,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":26.887,"5min":35.935000000000002,"15min":35.935000000000002},"last":7.835},{"interface":"front","average":{"1min":13.198,"5min":15.188000000000001,"15min":15.536},"min":{"1min":8.0129999999999999,"5min":8.0129999999999999,"15min":8.0129999999999999},"max":{"1min":25.497,"5min":30.433,"15min":30.433},"last":15.382}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.345000000000001,"5min":13.98,"15min":16.010000000000002},"min":{"1min":7.1070000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":28.527999999999999,"5min":50.137999999999998,"15min":50.137999999999998},"last":48.207999999999998},{"interface":"front","average":{"1min":16.071999999999999,"5min":16.396999999999998,"15min":16.539000000000001},"min":{"1min":10.430999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":41.265000000000001,"5min":46.325000000000003,"15min":46.325000000000003},"last":49.110999999999997}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.713000000000001,"5min":19.062999999999999,"15min":20.108000000000001},"min":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":42.633000000000003,"5min":50.320999999999998,"15min":50.320999999999998},"last":16.719999999999999},{"interface":"front","average":{"1min":15.576000000000001,"5min":15.792999999999999,"15min":17.120999999999999},"min":{"1min":5.9619999999999997,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":29.931000000000001,"5min":44.323,"15min":44.323},"last":21.062000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":15.397,"5min":17.364999999999998,"15min":19.513000000000002},"min":{"1min":9.6890000000000001,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":44.640999999999998,"5min":104.834,"15min":104.834},"last":11.928000000000001},{"interface":"front","average":{"1min":17.795999999999999,"5min":20.042000000000002,"15min":20.542000000000002},"min":{"1min":11.724,"5min":11.308999999999999,"15min":11.308999999999999},"max":{"1min":43.287999999999997,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.239000000000001}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.751999999999999,"5min":18.164000000000001,"15min":19.350000000000001},"min":{"1min":8.6020000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":26.753,"5min":42.915999999999997,"15min":42.915999999999997},"last":22.164000000000001},{"interface":"front","average":{"1min":16.710999999999999,"5min":17.780000000000001,"15min":18.882000000000001},"min":{"1min":10.266999999999999,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":43.697000000000003,"5min":43.697000000000003,"15min":43.697000000000003},"last":17.829999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476798,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28164,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343676,"statfs":{"total":96636764160,"available":96607924224,"internally_reserved":0,"allocated":1163264,"data_stored":678369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":9.1620000000000008,"5min":10.975,"15min":12.778},"min":{"1min":6.5380000000000003,"5min":4.835,"15min":4.835},"max":{"1min":15.888,"5min":80.640000000000001,"15min":80.640000000000001},"last":14.946999999999999},{"interface":"front","average":{"1min":12.728,"5min":11.108000000000001,"15min":11.477},"min":{"1min":8.6950000000000003,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":20.853000000000002,"5min":25.390000000000001,"15min":25.390000000000001},"last":19.425000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":16.413,"15min":19.614999999999998},"min":{"1min":8.5640000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":19.486999999999998,"5min":100.703,"15min":100.703},"last":16.443999999999999},{"interface":"front","average":{"1min":9.4510000000000005,"5min":14.611000000000001,"15min":17.943000000000001},"min":{"1min":5.4020000000000001,"5min":4.867,"15min":4.867},"max":{"1min":17.934999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":12.535}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":16.853000000000002,"15min":18.071000000000002},"min":{"1min":10.534000000000001,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":31.481999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":20.613},{"interface":"front","average":{"1min":12.157,"5min":11.869,"15min":13.27},"min":{"1min":5.9550000000000001,"5min":5.9550000000000001,"15min":5.9550000000000001},"max":{"1min":30.033000000000001,"5min":46.613,"15min":46.613},"last":11.285}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.465,"5min":15.667999999999999,"15min":17.754999999999999},"min":{"1min":6.6020000000000003,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":27.597000000000001,"5min":81.838999999999999,"15min":81.838999999999999},"last":21.786999999999999},{"interface":"front","average":{"1min":17.969000000000001,"5min":18.931999999999999,"15min":20.847000000000001},"min":{"1min":12.384,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":25.138000000000002,"5min":66.515000000000001,"15min":66.515000000000001},"last":23.710999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.042,"5min":14.403,"15min":15.888999999999999},"min":{"1min":6.4020000000000001,"5min":6.351,"15min":6.351},"max":{"1min":16.806999999999999,"5min":56.235999999999997,"15min":56.235999999999997},"last":22.91},{"interface":"front","average":{"1min":12.945,"5min":17.241,"15min":19.077000000000002},"min":{"1min":7.1680000000000001,"5min":7.1680000000000001,"15min":7.1680000000000001},"max":{"1min":31.472999999999999,"5min":82.381,"15min":82.381},"last":9.2780000000000005}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":14.901,"5min":16.239000000000001,"15min":18.032},"min":{"1min":8.5329999999999995,"5min":8.5329999999999995,"15min":8.5329999999999995},"max":{"1min":31.954999999999998,"5min":58.619,"15min":58.619},"last":25.073},{"interface":"front","average":{"1min":15.853,"5min":18.738,"15min":20.597000000000001},"min":{"1min":8.6359999999999992,"5min":7.8529999999999998,"15min":7.8529999999999998},"max":{"1min":33.671999999999997,"5min":83.632000000000005,"15min":83.632000000000005},"last":13.76}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":18.228999999999999,"5min":20.475999999999999,"15min":22.181999999999999},"min":{"1min":13.673,"5min":13.016,"15min":13.016},"max":{"1min":27.407,"5min":62.006999999999998,"15min":62.006999999999998},"last":25.765999999999998},{"interface":"front","average":{"1min":14.981,"5min":17.922000000000001,"15min":19.972000000000001},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":22.795000000000002,"5min":60.698999999999998,"15min":60.698999999999998},"last":17.884}]}]},{"osd":0,"up_from":16,"seq":68719476799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":289537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":10.026999999999999,"5min":14.167,"15min":18.539000000000001},"min":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":5.0049999999999999},"max":{"1min":24.733000000000001,"5min":252.88399999999999,"15min":252.88399999999999},"last":7.0940000000000003},{"interface":"front","average":{"1min":10.260999999999999,"5min":14.552,"15min":19.442},"min":{"1min":5.9409999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":17.584,"5min":274.91500000000002,"15min":274.91500000000002},"last":13.968999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":13.33,"5min":14.468,"15min":16.663},"min":{"1min":6.3319999999999999,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":35.383000000000003,"5min":82.866,"15min":82.866},"last":15.039999999999999},{"interface":"front","average":{"1min":14.218,"5min":15.249000000000001,"15min":16.135999999999999},"min":{"1min":6.6980000000000004,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":34.289999999999999,"5min":85.991,"15min":85.991},"last":11.483000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":11.598000000000001,"5min":12.791,"15min":14.571},"min":{"1min":7.1109999999999998,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":41.045000000000002,"5min":76.593000000000004,"15min":76.593000000000004},"last":16.649999999999999},{"interface":"front","average":{"1min":16.559999999999999,"5min":17.657,"15min":19.928000000000001},"min":{"1min":8.875,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":42.284999999999997,"5min":85.162000000000006,"15min":85.162000000000006},"last":19.265999999999998}]},{"osd":4,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":14.983000000000001,"5min":16.558,"15min":17.238},"min":{"1min":6.6230000000000002,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":31.988,"5min":76.387,"15min":76.387},"last":17.579000000000001},{"interface":"front","average":{"1min":12.487,"5min":12.813000000000001,"15min":13.67},"min":{"1min":5.8390000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":20.733000000000001,"5min":70.085999999999999,"15min":70.085999999999999},"last":29.375}]},{"osd":5,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":15.843999999999999,"5min":21.594000000000001,"15min":26.113},"min":{"1min":10.775,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":27.036000000000001,"5min":144.60499999999999,"15min":144.60499999999999},"last":35.901000000000003},{"interface":"front","average":{"1min":16.539999999999999,"5min":21.242999999999999,"15min":24.882999999999999},"min":{"1min":11.016,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":38.125,"5min":142.25,"15min":142.25},"last":30.295000000000002}]},{"osd":6,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":17.094999999999999,"5min":16.539000000000001,"15min":17.803000000000001},"min":{"1min":9.2729999999999997,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.847999999999999,"5min":75.471999999999994,"15min":75.471999999999994},"last":31.550000000000001},{"interface":"front","average":{"1min":15.427,"5min":19.684000000000001,"15min":22.881},"min":{"1min":7.8760000000000003,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":29.997,"5min":84.617000000000004,"15min":84.617000000000004},"last":37.479999999999997}]},{"osd":7,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":18.831,"5min":20.341000000000001,"15min":21.600999999999999},"min":{"1min":10.420999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":41.003,"5min":84.031000000000006,"15min":84.031000000000006},"last":34.926000000000002},{"interface":"front","average":{"1min":15.169,"5min":16.818999999999999,"15min":18.481000000000002},"min":{"1min":8.3629999999999995,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":27.341999999999999,"5min":77.974000000000004,"15min":77.974000000000004},"last":38.658999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509503,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":298537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.013,"5min":8.7430000000000003,"15min":9.3930000000000007},"min":{"1min":4.6109999999999998,"5min":4.234,"15min":4.234},"max":{"1min":48.454000000000001,"5min":48.454000000000001,"15min":48.454000000000001},"last":6.1200000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":10.225,"15min":10.269},"min":{"1min":5.3840000000000003,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":44.947000000000003,"5min":44.947000000000003,"15min":44.947000000000003},"last":10.108000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.641999999999999,"5min":10.75,"15min":11.603999999999999},"min":{"1min":5.8879999999999999,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":34.880000000000003,"5min":35.524000000000001,"15min":35.524000000000001},"last":8.8100000000000005},{"interface":"front","average":{"1min":11.723000000000001,"5min":12.821,"15min":13.641999999999999},"min":{"1min":7.1970000000000001,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":26.003,"5min":31.027000000000001,"15min":31.027000000000001},"last":14.218999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.923999999999999,"5min":12.789999999999999,"15min":12.273999999999999},"min":{"1min":7.1379999999999999,"5min":7.1379999999999999,"15min":7.1379999999999999},"max":{"1min":24.038,"5min":27.353000000000002,"15min":27.353000000000002},"last":12.166},{"interface":"front","average":{"1min":14.106,"5min":14.686999999999999,"15min":14.646000000000001},"min":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":24.634,"5min":32.915999999999997,"15min":32.915999999999997},"last":20.657}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.82,"5min":12.065,"15min":11.678000000000001},"min":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"max":{"1min":28.991,"5min":28.991,"15min":28.991},"last":13.335000000000001},{"interface":"front","average":{"1min":15.269,"5min":14.705,"15min":14.278},"min":{"1min":10.734,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":35.540999999999997,"5min":35.540999999999997,"15min":35.540999999999997},"last":20.062000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.009,"15min":17.068999999999999},"min":{"1min":10.19,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":27.757000000000001,"5min":53.899999999999999,"15min":53.899999999999999},"last":23.805},{"interface":"front","average":{"1min":14.430999999999999,"5min":17.372,"15min":17.57},"min":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"max":{"1min":19.631,"5min":55.914000000000001,"15min":55.914000000000001},"last":26.683}]},{"osd":6,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.116,"5min":18.611999999999998,"15min":21.172999999999998},"min":{"1min":9.423,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":20.387,"5min":93.073999999999998,"15min":93.073999999999998},"last":22.555},{"interface":"front","average":{"1min":16.431000000000001,"5min":15.773,"15min":17.733000000000001},"min":{"1min":6.7779999999999996,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":35.869999999999997,"5min":91.075999999999993,"15min":91.075999999999993},"last":31.016999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":17.501999999999999,"5min":16.015999999999998,"15min":16.25},"min":{"1min":10.579000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":31.844000000000001,"5min":33.988999999999997,"15min":33.988999999999997},"last":28.984000000000002},{"interface":"front","average":{"1min":17.699999999999999,"5min":17.460999999999999,"15min":16.974},"min":{"1min":11.65,"5min":10.885,"15min":10.885},"max":{"1min":37.154000000000003,"5min":37.154000000000003,"15min":37.154000000000003},"last":25.716999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":298537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.220000000000001,"5min":9.7940000000000005,"15min":10.138},"min":{"1min":4.718,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":35.454000000000001,"5min":70.965000000000003,"15min":70.965000000000003},"last":6.298},{"interface":"front","average":{"1min":8.1180000000000003,"5min":10.308,"15min":11.661},"min":{"1min":5.3979999999999997,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":17.193999999999999,"5min":73.582999999999998,"15min":73.582999999999998},"last":11.404}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":12.234999999999999,"5min":15.962999999999999,"15min":18.359999999999999},"min":{"1min":7.9580000000000002,"5min":4.383,"15min":4.383},"max":{"1min":22.164000000000001,"5min":217.595,"15min":217.595},"last":26.413},{"interface":"front","average":{"1min":13.619999999999999,"5min":17.018999999999998,"15min":21.198},"min":{"1min":7.0730000000000004,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":37.552,"5min":284.96499999999997,"15min":284.96499999999997},"last":12.007999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.404999999999999,"5min":13.512,"15min":13.757999999999999},"min":{"1min":6.7140000000000004,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.434999999999999,"5min":46.024999999999999,"15min":46.024999999999999},"last":13.693},{"interface":"front","average":{"1min":15.532,"5min":15.390000000000001,"15min":16.452000000000002},"min":{"1min":8.4120000000000008,"5min":7.7610000000000001,"15min":7.7610000000000001},"max":{"1min":40.298999999999999,"5min":46.966999999999999,"15min":46.966999999999999},"last":9.9879999999999995}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":13.816000000000001,"5min":15.863,"15min":15.851000000000001},"min":{"1min":9.4450000000000003,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":21.167000000000002,"5min":65.719999999999999,"15min":65.719999999999999},"last":23.809999999999999},{"interface":"front","average":{"1min":18.998000000000001,"5min":19.789000000000001,"15min":22.832999999999998},"min":{"1min":11.177,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":41.137,"5min":189.84999999999999,"15min":189.84999999999999},"last":10.81}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.972999999999999,"5min":17.817,"15min":16.527999999999999},"min":{"1min":9.4909999999999997,"5min":9.3409999999999993,"15min":9.3409999999999993},"max":{"1min":42.905999999999999,"5min":66.349999999999994,"15min":66.349999999999994},"last":16.952000000000002},{"interface":"front","average":{"1min":16.096,"5min":15.782999999999999,"15min":15.013999999999999},"min":{"1min":11.175000000000001,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":25.131,"5min":39.520000000000003,"15min":39.520000000000003},"last":14.321999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.574000000000002,"5min":18.863,"15min":18.902999999999999},"min":{"1min":9.7530000000000001,"5min":9.7530000000000001,"15min":9.7530000000000001},"max":{"1min":41.997,"5min":50.945,"15min":50.945},"last":22.699999999999999},{"interface":"front","average":{"1min":16.722999999999999,"5min":14.596,"15min":14.446999999999999},"min":{"1min":7.1550000000000002,"5min":6.4960000000000004,"15min":6.4960000000000004},"max":{"1min":38.826000000000001,"5min":51.948,"15min":51.948},"last":7.585}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.364000000000001,"5min":17.041,"15min":16.187000000000001},"min":{"1min":12.323,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":52.875,"5min":52.875,"15min":52.875},"last":8.4939999999999998},{"interface":"front","average":{"1min":22.992999999999999,"5min":20.244,"15min":19.75},"min":{"1min":14.509,"5min":11.18,"15min":11.18},"max":{"1min":51.877000000000002,"5min":71.346999999999994,"15min":71.346999999999994},"last":25.645}]}]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1212416,"data_stored":696377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":9.4220000000000006,"5min":9.2490000000000006,"15min":9.1739999999999995},"min":{"1min":5.2039999999999997,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":24.896000000000001,"5min":26.117999999999999,"15min":26.117999999999999},"last":7.8650000000000002},{"interface":"front","average":{"1min":9.7729999999999997,"5min":9.5340000000000007,"15min":9.5999999999999996},"min":{"1min":5.5110000000000001,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":23.355,"5min":28.145,"15min":28.145},"last":5.4580000000000002}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":10.904999999999999,"5min":12.387,"15min":13.345000000000001},"min":{"1min":5.2389999999999999,"5min":4.8659999999999997,"15min":4.8659999999999997},"max":{"1min":20.106000000000002,"5min":67.347999999999999,"15min":67.347999999999999},"last":10.583},{"interface":"front","average":{"1min":9.968,"5min":12.228,"15min":12.582000000000001},"min":{"1min":5.343,"5min":5.343,"15min":5.343},"max":{"1min":19.221,"5min":39.052,"15min":39.052},"last":13.752000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.557,"5min":12.074999999999999,"15min":12.106999999999999},"min":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"max":{"1min":24.247,"5min":31.873000000000001,"15min":31.873000000000001},"last":15.067},{"interface":"front","average":{"1min":14.055,"5min":14.414,"15min":15.505000000000001},"min":{"1min":7.6829999999999998,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":29.513999999999999,"5min":34.585000000000001,"15min":34.585000000000001},"last":20.064}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":14.154999999999999,"5min":14.811,"15min":15.292999999999999},"min":{"1min":8.4160000000000004,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":28.643000000000001,"5min":30.355,"15min":30.355},"last":8.5570000000000004},{"interface":"front","average":{"1min":15.125,"5min":14.696999999999999,"15min":15.029999999999999},"min":{"1min":9.0920000000000005,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":30.988,"5min":31.792000000000002,"15min":31.792000000000002},"last":12.523}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.83,"5min":20.387,"15min":26.422999999999998},"min":{"1min":10.773999999999999,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":24.518000000000001,"5min":334.86500000000001,"15min":334.86500000000001},"last":13.221},{"interface":"front","average":{"1min":13.234999999999999,"5min":20.062000000000001,"15min":26.206},"min":{"1min":7.5810000000000004,"5min":7.431,"15min":7.431},"max":{"1min":20.047999999999998,"5min":331.57900000000001,"15min":331.57900000000001},"last":11.365}]},{"osd":6,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.187999999999999,"5min":18.253,"15min":19.466999999999999},"min":{"1min":10.048,"5min":9.1310000000000002,"15min":9.1310000000000002},"max":{"1min":35.459000000000003,"5min":46.034999999999997,"15min":46.034999999999997},"last":19.030999999999999},{"interface":"front","average":{"1min":15.515000000000001,"5min":17.102,"15min":18.291},"min":{"1min":8.343,"5min":8.343,"15min":8.343},"max":{"1min":34.116,"5min":35.680999999999997,"15min":35.680999999999997},"last":20.835999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.087,"5min":21.350000000000001,"15min":28},"min":{"1min":11.781000000000001,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":19.779,"5min":332.322,"15min":332.322},"last":14.438000000000001},{"interface":"front","average":{"1min":19.039999999999999,"5min":24.308,"15min":31.135999999999999},"min":{"1min":13.331,"5min":10.467000000000001,"15min":10.467000000000001},"max":{"1min":26.495999999999999,"5min":325.93599999999998,"15min":325.93599999999998},"last":21.789000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":294033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.3860000000000001,"5min":8.7629999999999999,"15min":9.2940000000000005},"min":{"1min":4.7590000000000003,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.015000000000001,"5min":20.558,"15min":20.558},"last":26.225000000000001},{"interface":"front","average":{"1min":11.315,"5min":11.834,"15min":11.933999999999999},"min":{"1min":6.5359999999999996,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":17.52,"5min":20.893999999999998,"15min":20.893999999999998},"last":22.899000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.4,"5min":14.412000000000001,"15min":16.027999999999999},"min":{"1min":7.2130000000000001,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":18.568999999999999,"5min":51.890000000000001,"15min":51.890000000000001},"last":8.734},{"interface":"front","average":{"1min":7.6219999999999999,"5min":8.7490000000000006,"15min":8.5109999999999992},"min":{"1min":5.6360000000000001,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":14.318,"5min":37.042999999999999,"15min":37.042999999999999},"last":6.0289999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.321999999999999,"5min":12.026,"15min":12.414},"min":{"1min":4.359,"5min":4.359,"15min":4.359},"max":{"1min":20.170999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":27.832999999999998},{"interface":"front","average":{"1min":11.811999999999999,"5min":11.840999999999999,"15min":11.384},"min":{"1min":7.7549999999999999,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":18.934999999999999,"5min":35.002000000000002,"15min":35.002000000000002},"last":24.803999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.914999999999999,"5min":19.931999999999999,"15min":26.753},"min":{"1min":7.9039999999999999,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":15.723000000000001,"5min":228.17099999999999,"15min":228.17099999999999},"last":14.135},{"interface":"front","average":{"1min":14.678000000000001,"5min":22.178999999999998,"15min":29.501000000000001},"min":{"1min":9.9580000000000002,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":18.475000000000001,"5min":270.13,"15min":270.13},"last":17.843}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.047000000000001,"5min":14.993,"15min":16.600999999999999},"min":{"1min":10.432,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":16.914999999999999,"5min":86.980999999999995,"15min":86.980999999999995},"last":15.579000000000001},{"interface":"front","average":{"1min":14.48,"5min":17.702000000000002,"15min":19.783999999999999},"min":{"1min":9.2579999999999991,"5min":7.8719999999999999,"15min":7.8719999999999999},"max":{"1min":18.155000000000001,"5min":99.021000000000001,"15min":99.021000000000001},"last":14.782}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.275,"5min":19.556999999999999,"15min":21.327000000000002},"min":{"1min":11.198,"5min":10.254,"15min":10.254},"max":{"1min":19.477,"5min":97.191000000000003,"15min":97.191000000000003},"last":20.234000000000002},{"interface":"front","average":{"1min":16.013000000000002,"5min":18.779,"15min":20.585999999999999},"min":{"1min":10.664,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":19.811,"5min":64.048000000000002,"15min":64.048000000000002},"last":18.539000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":14.233000000000001,"5min":16.07,"15min":17.100000000000001},"min":{"1min":9.3729999999999993,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":18.184000000000001,"5min":74.064999999999998,"15min":74.064999999999998},"last":17.215},{"interface":"front","average":{"1min":16.649000000000001,"5min":20.157,"15min":21.806000000000001},"min":{"1min":10.581,"5min":10.055,"15min":10.055},"max":{"1min":20.405000000000001,"5min":92.477999999999994,"15min":92.477999999999994},"last":21.704000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:02:14.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:02:14.976 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:14.975+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:14.987 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:14.988 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:14.988 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:14.988 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:14.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:02:14.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:15.083 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:15.082+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:15.178 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:15.177+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:15.178 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:15.179 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:15.179 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:15.179 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:15.179 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:15.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:15.282 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:15.282+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:15.380 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:15.380+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:15.418 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:15.418 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:15.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 295.28635717499998, 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.424 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.81524372199999, 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:02:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:02:15.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.58505134299998, 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.430 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:02:15.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:02:15.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:02:15.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:02:15.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:15.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.58434379400001, 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:02:15.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.58370680000002, 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:15.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:02:15.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:02:15.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.52664160500001, 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:02:15.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.52515083399999, 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.453 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:02:15.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.49280117199999, 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:02:15.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:02:15.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.48617039700002, 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:02:15.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:02:15.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:02:15.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:02:15.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.485519716, 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:15.475 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.48204702499999, 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.48090830299998, 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:15.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:02:15.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:02:15.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:02:15.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.47996060600002, 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.486 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.451596612, 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:02:15.491 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:15.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:02:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:02:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.42338709500001, 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 286.81318916100003, 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:02:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 286.81254432100002, 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 286.79207756699998, 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:02:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:02:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:02:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 286.79132865100001, 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 286.79068658199998, 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:02:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:02:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:02:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:15.528 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:15.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:15.530 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:15.528+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:15.558 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:02:15.572 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2023-12-06T08:02:14.194970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223160,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751560,"statfs":{"total":773094113280,"available":772865597440,"internally_reserved":0,"allocated":7143424,"data_stored":3129952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.344274"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.099848+0000","last_change":"2023-12-06T08:01:15.480948+0000","last_active":"2023-12-06T08:02:09.099848+0000","last_peered":"2023-12-06T08:02:09.099848+0000","last_clean":"2023-12-06T08:02:09.099848+0000","last_became_active":"2023-12-06T08:01:15.474602+0000","last_became_peered":"2023-12-06T08:01:15.474602+0000","last_unstale":"2023-12-06T08:02:09.099848+0000","last_undegraded":"2023-12-06T08:02:09.099848+0000","last_fullsized":"2023-12-06T08:02:09.099848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:07:38.010954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.440024+0000","last_change":"2023-12-06T07:58:03.120993+0000","last_active":"2023-12-06T08:02:09.440024+0000","last_peered":"2023-12-06T08:02:09.440024+0000","last_clean":"2023-12-06T08:02:09.440024+0000","last_became_active":"2023-12-06T07:58:03.114274+0000","last_became_peered":"2023-12-06T07:58:03.114274+0000","last_unstale":"2023-12-06T08:02:09.440024+0000","last_undegraded":"2023-12-06T08:02:09.440024+0000","last_fullsized":"2023-12-06T08:02:09.440024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:32.901237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.092542+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:02:09.092542+0000","last_peered":"2023-12-06T08:02:09.092542+0000","last_clean":"2023-12-06T08:02:09.092542+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:02:09.092542+0000","last_undegraded":"2023-12-06T08:02:09.092542+0000","last_fullsized":"2023-12-06T08:02:09.092542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.791210+0000","last_change":"2023-12-06T07:57:30.145051+0000","last_active":"2023-12-06T08:02:09.791210+0000","last_peered":"2023-12-06T08:02:09.791210+0000","last_clean":"2023-12-06T08:02:09.791210+0000","last_became_active":"2023-12-06T07:57:30.119439+0000","last_became_peered":"2023-12-06T07:57:30.119439+0000","last_unstale":"2023-12-06T08:02:09.791210+0000","last_undegraded":"2023-12-06T08:02:09.791210+0000","last_fullsized":"2023-12-06T08:02:09.791210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:12:26.522660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:08.226461+0000","last_change":"2023-12-06T08:01:04.450104+0000","last_active":"2023-12-06T08:02:08.226461+0000","last_peered":"2023-12-06T08:02:08.226461+0000","last_clean":"2023-12-06T08:02:08.226461+0000","last_became_active":"2023-12-06T08:01:04.441340+0000","last_became_peered":"2023-12-06T08:01:04.441340+0000","last_unstale":"2023-12-06T08:02:08.226461+0000","last_undegraded":"2023-12-06T08:02:08.226461+0000","last_fullsized":"2023-12-06T08:02:08.226461+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:26:50.232248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.436089+0000","last_change":"2023-12-06T07:57:29.840367+0000","last_active":"2023-12-06T08:02:09.436089+0000","last_peered":"2023-12-06T08:02:09.436089+0000","last_clean":"2023-12-06T08:02:09.436089+0000","last_became_active":"2023-12-06T07:57:29.812796+0000","last_became_peered":"2023-12-06T07:57:29.812796+0000","last_unstale":"2023-12-06T08:02:09.436089+0000","last_undegraded":"2023-12-06T08:02:09.436089+0000","last_fullsized":"2023-12-06T08:02:09.436089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:34:03.154657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2023-12-06T08:02:12.773593+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:02:12.773593+0000","last_peered":"2023-12-06T08:02:12.773593+0000","last_clean":"2023-12-06T08:02:12.773593+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:02:12.773593+0000","last_undegraded":"2023-12-06T08:02:12.773593+0000","last_fullsized":"2023-12-06T08:02:12.773593+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.200780+0000","last_change":"2023-12-06T08:01:59.482848+0000","last_active":"2023-12-06T08:02:09.200780+0000","last_peered":"2023-12-06T08:02:09.200780+0000","last_clean":"2023-12-06T08:02:09.200780+0000","last_became_active":"2023-12-06T08:01:59.452600+0000","last_became_peered":"2023-12-06T08:01:59.452600+0000","last_unstale":"2023-12-06T08:02:09.200780+0000","last_undegraded":"2023-12-06T08:02:09.200780+0000","last_fullsized":"2023-12-06T08:02:09.200780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:39:53.539482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":171,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-12-06T08:02:09.088553+0000","last_change":"2023-12-06T07:57:20.594093+0000","last_active":"2023-12-06T08:02:09.088553+0000","last_peered":"2023-12-06T08:02:09.088553+0000","last_clean":"2023-12-06T08:02:09.088553+0000","last_became_active":"2023-12-06T07:57:20.562969+0000","last_became_peered":"2023-12-06T07:57:20.562969+0000","last_unstale":"2023-12-06T08:02:09.088553+0000","last_undegraded":"2023-12-06T08:02:09.088553+0000","last_fullsized":"2023-12-06T08:02:09.088553+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:11:29.916506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476797,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":294033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":9.0860000000000003,"15min":9.4220000000000006},"min":{"1min":4.1100000000000003,"5min":4.1100000000000003,"15min":4.1100000000000003},"max":{"1min":16.015000000000001,"5min":27.638999999999999,"15min":27.638999999999999},"last":8.1229999999999993},{"interface":"front","average":{"1min":8.9239999999999995,"5min":8.8369999999999997,"15min":8.6850000000000005},"min":{"1min":5.008,"5min":4.8029999999999999,"15min":4.8029999999999999},"max":{"1min":13.74,"5min":24.085000000000001,"15min":24.085000000000001},"last":8.6159999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.744999999999999,"5min":11.625,"15min":12.619},"min":{"1min":6.6299999999999999,"5min":6.1219999999999999,"15min":6.1219999999999999},"max":{"1min":28.713999999999999,"5min":29.213000000000001,"15min":29.213000000000001},"last":18.27},{"interface":"front","average":{"1min":11.446999999999999,"5min":13.115,"15min":14.191000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":30.204000000000001,"5min":39.337000000000003,"15min":39.337000000000003},"last":9.9480000000000004}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.166,"5min":12.4,"15min":12.557},"min":{"1min":7.3070000000000004,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":27.027000000000001,"5min":27.861000000000001,"15min":27.861000000000001},"last":10.521000000000001},{"interface":"front","average":{"1min":12.935,"5min":13.534000000000001,"15min":13.991},"min":{"1min":9.1110000000000007,"5min":6.7569999999999997,"15min":6.7569999999999997},"max":{"1min":21.375,"5min":26.388000000000002,"15min":26.388000000000002},"last":11.228}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.247,"5min":14.007999999999999,"15min":13.446},"min":{"1min":8.3350000000000009,"5min":7.8609999999999998,"15min":7.8609999999999998},"max":{"1min":18.373999999999999,"5min":27.263000000000002,"15min":27.263000000000002},"last":12.381},{"interface":"front","average":{"1min":14.763999999999999,"5min":15.885999999999999,"15min":16.044},"min":{"1min":10.791,"5min":7.806,"15min":7.806},"max":{"1min":25.434000000000001,"5min":54.225999999999999,"15min":54.225999999999999},"last":17.497}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.474,"5min":16.591999999999999,"15min":17.013000000000002},"min":{"1min":7.742,"5min":7.742,"15min":7.742},"max":{"1min":23.84,"5min":50.066000000000003,"15min":50.066000000000003},"last":14.228999999999999},{"interface":"front","average":{"1min":16.983000000000001,"5min":17.257000000000001,"15min":17.350000000000001},"min":{"1min":11.358000000000001,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":25.030999999999999,"5min":50.478000000000002,"15min":50.478000000000002},"last":16.597999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":16.305,"5min":14.528,"15min":14.744},"min":{"1min":9.1760000000000002,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":23.544,"5min":27.890000000000001,"15min":27.890000000000001},"last":13.640000000000001},{"interface":"front","average":{"1min":17.661999999999999,"5min":18.466000000000001,"15min":18.699000000000002},"min":{"1min":13,"5min":9.4299999999999997,"15min":9.4299999999999997},"max":{"1min":24.808,"5min":44.255000000000003,"15min":44.255000000000003},"last":19.170999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.577999999999999,"5min":17.204999999999998,"15min":18.709},"min":{"1min":8.1319999999999997,"5min":8.0329999999999995,"15min":8.0329999999999995},"max":{"1min":17.728000000000002,"5min":40.832000000000001,"15min":40.832000000000001},"last":15.786},{"interface":"front","average":{"1min":16.411999999999999,"5min":17.524000000000001,"15min":18.338999999999999},"min":{"1min":9.9190000000000005,"5min":9.2330000000000005,"15min":9.2330000000000005},"max":{"1min":25.026,"5min":39.331000000000003,"15min":39.331000000000003},"last":14.85}]}]},{"osd":2,"up_from":16,"seq":68719476798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":280529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.478999999999999,"5min":9.8840000000000003,"15min":10.292},"min":{"1min":4.548,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":23.946999999999999},"last":6.6180000000000003},{"interface":"front","average":{"1min":10.711,"5min":10.263999999999999,"15min":10.494999999999999},"min":{"1min":6.9610000000000003,"5min":5.0289999999999999,"15min":5.0289999999999999},"max":{"1min":31.690000000000001,"5min":34.981000000000002,"15min":34.981000000000002},"last":10.526}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.083,"5min":16.556000000000001,"15min":20.513999999999999},"min":{"1min":6.4059999999999997,"5min":6.4059999999999997,"15min":6.4059999999999997},"max":{"1min":18.251000000000001,"5min":165.68600000000001,"15min":165.68600000000001},"last":13.292999999999999},{"interface":"front","average":{"1min":13.439,"5min":16.741,"15min":21.100999999999999},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":19.774000000000001,"5min":153.68199999999999,"15min":153.68199999999999},"last":14.071}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.566000000000001,"5min":10.433999999999999,"15min":10.992000000000001},"min":{"1min":6.5579999999999998,"5min":5.4299999999999997,"15min":5.4299999999999997},"max":{"1min":26.887,"5min":35.935000000000002,"15min":35.935000000000002},"last":7.835},{"interface":"front","average":{"1min":13.198,"5min":15.188000000000001,"15min":15.536},"min":{"1min":8.0129999999999999,"5min":8.0129999999999999,"15min":8.0129999999999999},"max":{"1min":25.497,"5min":30.433,"15min":30.433},"last":15.382}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.345000000000001,"5min":13.98,"15min":16.010000000000002},"min":{"1min":7.1070000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":28.527999999999999,"5min":50.137999999999998,"15min":50.137999999999998},"last":48.207999999999998},{"interface":"front","average":{"1min":16.071999999999999,"5min":16.396999999999998,"15min":16.539000000000001},"min":{"1min":10.430999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":41.265000000000001,"5min":46.325000000000003,"15min":46.325000000000003},"last":49.110999999999997}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.713000000000001,"5min":19.062999999999999,"15min":20.108000000000001},"min":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":42.633000000000003,"5min":50.320999999999998,"15min":50.320999999999998},"last":16.719999999999999},{"interface":"front","average":{"1min":15.576000000000001,"5min":15.792999999999999,"15min":17.120999999999999},"min":{"1min":5.9619999999999997,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":29.931000000000001,"5min":44.323,"15min":44.323},"last":21.062000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":15.397,"5min":17.364999999999998,"15min":19.513000000000002},"min":{"1min":9.6890000000000001,"5min":8.6080000000000005,"15min":8.6080000000000005},"max":{"1min":44.640999999999998,"5min":104.834,"15min":104.834},"last":11.928000000000001},{"interface":"front","average":{"1min":17.795999999999999,"5min":20.042000000000002,"15min":20.542000000000002},"min":{"1min":11.724,"5min":11.308999999999999,"15min":11.308999999999999},"max":{"1min":43.287999999999997,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.239000000000001}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":17.751999999999999,"5min":18.164000000000001,"15min":19.350000000000001},"min":{"1min":8.6020000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":26.753,"5min":42.915999999999997,"15min":42.915999999999997},"last":22.164000000000001},{"interface":"front","average":{"1min":16.710999999999999,"5min":17.780000000000001,"15min":18.882000000000001},"min":{"1min":10.266999999999999,"5min":9.8239999999999998,"15min":9.8239999999999998},"max":{"1min":43.697000000000003,"5min":43.697000000000003,"15min":43.697000000000003},"last":17.829999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476798,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28164,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343676,"statfs":{"total":96636764160,"available":96607924224,"internally_reserved":0,"allocated":1163264,"data_stored":678369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":9.1620000000000008,"5min":10.975,"15min":12.778},"min":{"1min":6.5380000000000003,"5min":4.835,"15min":4.835},"max":{"1min":15.888,"5min":80.640000000000001,"15min":80.640000000000001},"last":14.946999999999999},{"interface":"front","average":{"1min":12.728,"5min":11.108000000000001,"15min":11.477},"min":{"1min":8.6950000000000003,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":20.853000000000002,"5min":25.390000000000001,"15min":25.390000000000001},"last":19.425000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.156000000000001,"5min":16.413,"15min":19.614999999999998},"min":{"1min":8.5640000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":19.486999999999998,"5min":100.703,"15min":100.703},"last":16.443999999999999},{"interface":"front","average":{"1min":9.4510000000000005,"5min":14.611000000000001,"15min":17.943000000000001},"min":{"1min":5.4020000000000001,"5min":4.867,"15min":4.867},"max":{"1min":17.934999999999999,"5min":95.960999999999999,"15min":95.960999999999999},"last":12.535}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":16.853000000000002,"15min":18.071000000000002},"min":{"1min":10.534000000000001,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":31.481999999999999,"5min":42.167999999999999,"15min":42.167999999999999},"last":20.613},{"interface":"front","average":{"1min":12.157,"5min":11.869,"15min":13.27},"min":{"1min":5.9550000000000001,"5min":5.9550000000000001,"15min":5.9550000000000001},"max":{"1min":30.033000000000001,"5min":46.613,"15min":46.613},"last":11.285}]},{"osd":4,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.465,"5min":15.667999999999999,"15min":17.754999999999999},"min":{"1min":6.6020000000000003,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":27.597000000000001,"5min":81.838999999999999,"15min":81.838999999999999},"last":21.786999999999999},{"interface":"front","average":{"1min":17.969000000000001,"5min":18.931999999999999,"15min":20.847000000000001},"min":{"1min":12.384,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":25.138000000000002,"5min":66.515000000000001,"15min":66.515000000000001},"last":23.710999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.042,"5min":14.403,"15min":15.888999999999999},"min":{"1min":6.4020000000000001,"5min":6.351,"15min":6.351},"max":{"1min":16.806999999999999,"5min":56.235999999999997,"15min":56.235999999999997},"last":22.91},{"interface":"front","average":{"1min":12.945,"5min":17.241,"15min":19.077000000000002},"min":{"1min":7.1680000000000001,"5min":7.1680000000000001,"15min":7.1680000000000001},"max":{"1min":31.472999999999999,"5min":82.381,"15min":82.381},"last":9.2780000000000005}]},{"osd":6,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":14.901,"5min":16.239000000000001,"15min":18.032},"min":{"1min":8.5329999999999995,"5min":8.5329999999999995,"15min":8.5329999999999995},"max":{"1min":31.954999999999998,"5min":58.619,"15min":58.619},"last":25.073},{"interface":"front","average":{"1min":15.853,"5min":18.738,"15min":20.597000000000001},"min":{"1min":8.6359999999999992,"5min":7.8529999999999998,"15min":7.8529999999999998},"max":{"1min":33.671999999999997,"5min":83.632000000000005,"15min":83.632000000000005},"last":13.76}]},{"osd":7,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":18.228999999999999,"5min":20.475999999999999,"15min":22.181999999999999},"min":{"1min":13.673,"5min":13.016,"15min":13.016},"max":{"1min":27.407,"5min":62.006999999999998,"15min":62.006999999999998},"last":25.765999999999998},{"interface":"front","average":{"1min":14.981,"5min":17.922000000000001,"15min":19.972000000000001},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":22.795000000000002,"5min":60.698999999999998,"15min":60.698999999999998},"last":17.884}]}]},{"osd":0,"up_from":16,"seq":68719476799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":289537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":10.026999999999999,"5min":14.167,"15min":18.539000000000001},"min":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":5.0049999999999999},"max":{"1min":24.733000000000001,"5min":252.88399999999999,"15min":252.88399999999999},"last":7.0940000000000003},{"interface":"front","average":{"1min":10.260999999999999,"5min":14.552,"15min":19.442},"min":{"1min":5.9409999999999998,"5min":5.5810000000000004,"15min":5.5810000000000004},"max":{"1min":17.584,"5min":274.91500000000002,"15min":274.91500000000002},"last":13.968999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":13.33,"5min":14.468,"15min":16.663},"min":{"1min":6.3319999999999999,"5min":6.2649999999999997,"15min":6.2649999999999997},"max":{"1min":35.383000000000003,"5min":82.866,"15min":82.866},"last":15.039999999999999},{"interface":"front","average":{"1min":14.218,"5min":15.249000000000001,"15min":16.135999999999999},"min":{"1min":6.6980000000000004,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":34.289999999999999,"5min":85.991,"15min":85.991},"last":11.483000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":11.598000000000001,"5min":12.791,"15min":14.571},"min":{"1min":7.1109999999999998,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":41.045000000000002,"5min":76.593000000000004,"15min":76.593000000000004},"last":16.649999999999999},{"interface":"front","average":{"1min":16.559999999999999,"5min":17.657,"15min":19.928000000000001},"min":{"1min":8.875,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":42.284999999999997,"5min":85.162000000000006,"15min":85.162000000000006},"last":19.265999999999998}]},{"osd":4,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":14.983000000000001,"5min":16.558,"15min":17.238},"min":{"1min":6.6230000000000002,"5min":6.6230000000000002,"15min":6.6230000000000002},"max":{"1min":31.988,"5min":76.387,"15min":76.387},"last":17.579000000000001},{"interface":"front","average":{"1min":12.487,"5min":12.813000000000001,"15min":13.67},"min":{"1min":5.8390000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":20.733000000000001,"5min":70.085999999999999,"15min":70.085999999999999},"last":29.375}]},{"osd":5,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":15.843999999999999,"5min":21.594000000000001,"15min":26.113},"min":{"1min":10.775,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":27.036000000000001,"5min":144.60499999999999,"15min":144.60499999999999},"last":35.901000000000003},{"interface":"front","average":{"1min":16.539999999999999,"5min":21.242999999999999,"15min":24.882999999999999},"min":{"1min":11.016,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":38.125,"5min":142.25,"15min":142.25},"last":30.295000000000002}]},{"osd":6,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":17.094999999999999,"5min":16.539000000000001,"15min":17.803000000000001},"min":{"1min":9.2729999999999997,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.847999999999999,"5min":75.471999999999994,"15min":75.471999999999994},"last":31.550000000000001},{"interface":"front","average":{"1min":15.427,"5min":19.684000000000001,"15min":22.881},"min":{"1min":7.8760000000000003,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":29.997,"5min":84.617000000000004,"15min":84.617000000000004},"last":37.479999999999997}]},{"osd":7,"last update":"Wed Dec 6 08:01:30 2023","interfaces":[{"interface":"back","average":{"1min":18.831,"5min":20.341000000000001,"15min":21.600999999999999},"min":{"1min":10.420999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"max":{"1min":41.003,"5min":84.031000000000006,"15min":84.031000000000006},"last":34.926000000000002},{"interface":"front","average":{"1min":15.169,"5min":16.818999999999999,"15min":18.481000000000002},"min":{"1min":8.3629999999999995,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":27.341999999999999,"5min":77.974000000000004,"15min":77.974000000000004},"last":38.658999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509503,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":298537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.013,"5min":8.7430000000000003,"15min":9.3930000000000007},"min":{"1min":4.6109999999999998,"5min":4.234,"15min":4.234},"max":{"1min":48.454000000000001,"5min":48.454000000000001,"15min":48.454000000000001},"last":6.1200000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":10.225,"15min":10.269},"min":{"1min":5.3840000000000003,"5min":5.0960000000000001,"15min":5.0960000000000001},"max":{"1min":44.947000000000003,"5min":44.947000000000003,"15min":44.947000000000003},"last":10.108000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":10.641999999999999,"5min":10.75,"15min":11.603999999999999},"min":{"1min":5.8879999999999999,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":34.880000000000003,"5min":35.524000000000001,"15min":35.524000000000001},"last":8.8100000000000005},{"interface":"front","average":{"1min":11.723000000000001,"5min":12.821,"15min":13.641999999999999},"min":{"1min":7.1970000000000001,"5min":5.6269999999999998,"15min":5.6269999999999998},"max":{"1min":26.003,"5min":31.027000000000001,"15min":31.027000000000001},"last":14.218999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":12.923999999999999,"5min":12.789999999999999,"15min":12.273999999999999},"min":{"1min":7.1379999999999999,"5min":7.1379999999999999,"15min":7.1379999999999999},"max":{"1min":24.038,"5min":27.353000000000002,"15min":27.353000000000002},"last":12.166},{"interface":"front","average":{"1min":14.106,"5min":14.686999999999999,"15min":14.646000000000001},"min":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":24.634,"5min":32.915999999999997,"15min":32.915999999999997},"last":20.657}]},{"osd":3,"last update":"Wed Dec 6 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":13.82,"5min":12.065,"15min":11.678000000000001},"min":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"max":{"1min":28.991,"5min":28.991,"15min":28.991},"last":13.335000000000001},{"interface":"front","average":{"1min":15.269,"5min":14.705,"15min":14.278},"min":{"1min":10.734,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":35.540999999999997,"5min":35.540999999999997,"15min":35.540999999999997},"last":20.062000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.009,"15min":17.068999999999999},"min":{"1min":10.19,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":27.757000000000001,"5min":53.899999999999999,"15min":53.899999999999999},"last":23.805},{"interface":"front","average":{"1min":14.430999999999999,"5min":17.372,"15min":17.57},"min":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"max":{"1min":19.631,"5min":55.914000000000001,"15min":55.914000000000001},"last":26.683}]},{"osd":6,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":15.116,"5min":18.611999999999998,"15min":21.172999999999998},"min":{"1min":9.423,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":20.387,"5min":93.073999999999998,"15min":93.073999999999998},"last":22.555},{"interface":"front","average":{"1min":16.431000000000001,"5min":15.773,"15min":17.733000000000001},"min":{"1min":6.7779999999999996,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":35.869999999999997,"5min":91.075999999999993,"15min":91.075999999999993},"last":31.016999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:24 2023","interfaces":[{"interface":"back","average":{"1min":17.501999999999999,"5min":16.015999999999998,"15min":16.25},"min":{"1min":10.579000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"max":{"1min":31.844000000000001,"5min":33.988999999999997,"15min":33.988999999999997},"last":28.984000000000002},{"interface":"front","average":{"1min":17.699999999999999,"5min":17.460999999999999,"15min":16.974},"min":{"1min":11.65,"5min":10.885,"15min":10.885},"max":{"1min":37.154000000000003,"5min":37.154000000000003,"15min":37.154000000000003},"last":25.716999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":298537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.220000000000001,"5min":9.7940000000000005,"15min":10.138},"min":{"1min":4.718,"5min":4.0810000000000004,"15min":4.0810000000000004},"max":{"1min":35.454000000000001,"5min":70.965000000000003,"15min":70.965000000000003},"last":6.298},{"interface":"front","average":{"1min":8.1180000000000003,"5min":10.308,"15min":11.661},"min":{"1min":5.3979999999999997,"5min":4.6859999999999999,"15min":4.6859999999999999},"max":{"1min":17.193999999999999,"5min":73.582999999999998,"15min":73.582999999999998},"last":11.404}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":12.234999999999999,"5min":15.962999999999999,"15min":18.359999999999999},"min":{"1min":7.9580000000000002,"5min":4.383,"15min":4.383},"max":{"1min":22.164000000000001,"5min":217.595,"15min":217.595},"last":26.413},{"interface":"front","average":{"1min":13.619999999999999,"5min":17.018999999999998,"15min":21.198},"min":{"1min":7.0730000000000004,"5min":5.9859999999999998,"15min":5.9859999999999998},"max":{"1min":37.552,"5min":284.96499999999997,"15min":284.96499999999997},"last":12.007999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.404999999999999,"5min":13.512,"15min":13.757999999999999},"min":{"1min":6.7140000000000004,"5min":6.0469999999999997,"15min":6.0469999999999997},"max":{"1min":17.434999999999999,"5min":46.024999999999999,"15min":46.024999999999999},"last":13.693},{"interface":"front","average":{"1min":15.532,"5min":15.390000000000001,"15min":16.452000000000002},"min":{"1min":8.4120000000000008,"5min":7.7610000000000001,"15min":7.7610000000000001},"max":{"1min":40.298999999999999,"5min":46.966999999999999,"15min":46.966999999999999},"last":9.9879999999999995}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":13.816000000000001,"5min":15.863,"15min":15.851000000000001},"min":{"1min":9.4450000000000003,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":21.167000000000002,"5min":65.719999999999999,"15min":65.719999999999999},"last":23.809999999999999},{"interface":"front","average":{"1min":18.998000000000001,"5min":19.789000000000001,"15min":22.832999999999998},"min":{"1min":11.177,"5min":7.6139999999999999,"15min":7.6139999999999999},"max":{"1min":41.137,"5min":189.84999999999999,"15min":189.84999999999999},"last":10.81}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.972999999999999,"5min":17.817,"15min":16.527999999999999},"min":{"1min":9.4909999999999997,"5min":9.3409999999999993,"15min":9.3409999999999993},"max":{"1min":42.905999999999999,"5min":66.349999999999994,"15min":66.349999999999994},"last":16.952000000000002},{"interface":"front","average":{"1min":16.096,"5min":15.782999999999999,"15min":15.013999999999999},"min":{"1min":11.175000000000001,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":25.131,"5min":39.520000000000003,"15min":39.520000000000003},"last":14.321999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.574000000000002,"5min":18.863,"15min":18.902999999999999},"min":{"1min":9.7530000000000001,"5min":9.7530000000000001,"15min":9.7530000000000001},"max":{"1min":41.997,"5min":50.945,"15min":50.945},"last":22.699999999999999},{"interface":"front","average":{"1min":16.722999999999999,"5min":14.596,"15min":14.446999999999999},"min":{"1min":7.1550000000000002,"5min":6.4960000000000004,"15min":6.4960000000000004},"max":{"1min":38.826000000000001,"5min":51.948,"15min":51.948},"last":7.585}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":19.364000000000001,"5min":17.041,"15min":16.187000000000001},"min":{"1min":12.323,"5min":7.6740000000000004,"15min":7.6740000000000004},"max":{"1min":52.875,"5min":52.875,"15min":52.875},"last":8.4939999999999998},{"interface":"front","average":{"1min":22.992999999999999,"5min":20.244,"15min":19.75},"min":{"1min":14.509,"5min":11.18,"15min":11.18},"max":{"1min":51.877000000000002,"5min":71.346999999999994,"15min":71.346999999999994},"last":25.645}]}]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1212416,"data_stored":696377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":9.4220000000000006,"5min":9.2490000000000006,"15min":9.1739999999999995},"min":{"1min":5.2039999999999997,"5min":4.6470000000000002,"15min":4.6470000000000002},"max":{"1min":24.896000000000001,"5min":26.117999999999999,"15min":26.117999999999999},"last":7.8650000000000002},{"interface":"front","average":{"1min":9.7729999999999997,"5min":9.5340000000000007,"15min":9.5999999999999996},"min":{"1min":5.5110000000000001,"5min":4.6509999999999998,"15min":4.6509999999999998},"max":{"1min":23.355,"5min":28.145,"15min":28.145},"last":5.4580000000000002}]},{"osd":1,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":10.904999999999999,"5min":12.387,"15min":13.345000000000001},"min":{"1min":5.2389999999999999,"5min":4.8659999999999997,"15min":4.8659999999999997},"max":{"1min":20.106000000000002,"5min":67.347999999999999,"15min":67.347999999999999},"last":10.583},{"interface":"front","average":{"1min":9.968,"5min":12.228,"15min":12.582000000000001},"min":{"1min":5.343,"5min":5.343,"15min":5.343},"max":{"1min":19.221,"5min":39.052,"15min":39.052},"last":13.752000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":11.557,"5min":12.074999999999999,"15min":12.106999999999999},"min":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"max":{"1min":24.247,"5min":31.873000000000001,"15min":31.873000000000001},"last":15.067},{"interface":"front","average":{"1min":14.055,"5min":14.414,"15min":15.505000000000001},"min":{"1min":7.6829999999999998,"5min":6.8209999999999997,"15min":6.8209999999999997},"max":{"1min":29.513999999999999,"5min":34.585000000000001,"15min":34.585000000000001},"last":20.064}]},{"osd":3,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":14.154999999999999,"5min":14.811,"15min":15.292999999999999},"min":{"1min":8.4160000000000004,"5min":6.9450000000000003,"15min":6.9450000000000003},"max":{"1min":28.643000000000001,"5min":30.355,"15min":30.355},"last":8.5570000000000004},{"interface":"front","average":{"1min":15.125,"5min":14.696999999999999,"15min":15.029999999999999},"min":{"1min":9.0920000000000005,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":30.988,"5min":31.792000000000002,"15min":31.792000000000002},"last":12.523}]},{"osd":4,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.83,"5min":20.387,"15min":26.422999999999998},"min":{"1min":10.773999999999999,"5min":7.0949999999999998,"15min":7.0949999999999998},"max":{"1min":24.518000000000001,"5min":334.86500000000001,"15min":334.86500000000001},"last":13.221},{"interface":"front","average":{"1min":13.234999999999999,"5min":20.062000000000001,"15min":26.206},"min":{"1min":7.5810000000000004,"5min":7.431,"15min":7.431},"max":{"1min":20.047999999999998,"5min":331.57900000000001,"15min":331.57900000000001},"last":11.365}]},{"osd":6,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":18.187999999999999,"5min":18.253,"15min":19.466999999999999},"min":{"1min":10.048,"5min":9.1310000000000002,"15min":9.1310000000000002},"max":{"1min":35.459000000000003,"5min":46.034999999999997,"15min":46.034999999999997},"last":19.030999999999999},{"interface":"front","average":{"1min":15.515000000000001,"5min":17.102,"15min":18.291},"min":{"1min":8.343,"5min":8.343,"15min":8.343},"max":{"1min":34.116,"5min":35.680999999999997,"15min":35.680999999999997},"last":20.835999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:01:23 2023","interfaces":[{"interface":"back","average":{"1min":15.087,"5min":21.350000000000001,"15min":28},"min":{"1min":11.781000000000001,"5min":8.4239999999999995,"15min":8.4239999999999995},"max":{"1min":19.779,"5min":332.322,"15min":332.322},"last":14.438000000000001},{"interface":"front","average":{"1min":19.039999999999999,"5min":24.308,"15min":31.135999999999999},"min":{"1min":13.331,"5min":10.467000000000001,"15min":10.467000000000001},"max":{"1min":26.495999999999999,"5min":325.93599999999998,"15min":325.93599999999998},"last":21.789000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":294033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":7.3860000000000001,"5min":8.7629999999999999,"15min":9.2940000000000005},"min":{"1min":4.7590000000000003,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.015000000000001,"5min":20.558,"15min":20.558},"last":26.225000000000001},{"interface":"front","average":{"1min":11.315,"5min":11.834,"15min":11.933999999999999},"min":{"1min":6.5359999999999996,"5min":6.5099999999999998,"15min":6.5099999999999998},"max":{"1min":17.52,"5min":20.893999999999998,"15min":20.893999999999998},"last":22.899000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.4,"5min":14.412000000000001,"15min":16.027999999999999},"min":{"1min":7.2130000000000001,"5min":6.0890000000000004,"15min":6.0890000000000004},"max":{"1min":18.568999999999999,"5min":51.890000000000001,"15min":51.890000000000001},"last":8.734},{"interface":"front","average":{"1min":7.6219999999999999,"5min":8.7490000000000006,"15min":8.5109999999999992},"min":{"1min":5.6360000000000001,"5min":4.2110000000000003,"15min":4.2110000000000003},"max":{"1min":14.318,"5min":37.042999999999999,"15min":37.042999999999999},"last":6.0289999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":10.321999999999999,"5min":12.026,"15min":12.414},"min":{"1min":4.359,"5min":4.359,"15min":4.359},"max":{"1min":20.170999999999999,"5min":35.225999999999999,"15min":35.225999999999999},"last":27.832999999999998},{"interface":"front","average":{"1min":11.811999999999999,"5min":11.840999999999999,"15min":11.384},"min":{"1min":7.7549999999999999,"5min":5.7590000000000003,"15min":5.7590000000000003},"max":{"1min":18.934999999999999,"5min":35.002000000000002,"15min":35.002000000000002},"last":24.803999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":11.914999999999999,"5min":19.931999999999999,"15min":26.753},"min":{"1min":7.9039999999999999,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":15.723000000000001,"5min":228.17099999999999,"15min":228.17099999999999},"last":14.135},{"interface":"front","average":{"1min":14.678000000000001,"5min":22.178999999999998,"15min":29.501000000000001},"min":{"1min":9.9580000000000002,"5min":9.7769999999999992,"15min":9.7769999999999992},"max":{"1min":18.475000000000001,"5min":270.13,"15min":270.13},"last":17.843}]},{"osd":4,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":13.047000000000001,"5min":14.993,"15min":16.600999999999999},"min":{"1min":10.432,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":16.914999999999999,"5min":86.980999999999995,"15min":86.980999999999995},"last":15.579000000000001},{"interface":"front","average":{"1min":14.48,"5min":17.702000000000002,"15min":19.783999999999999},"min":{"1min":9.2579999999999991,"5min":7.8719999999999999,"15min":7.8719999999999999},"max":{"1min":18.155000000000001,"5min":99.021000000000001,"15min":99.021000000000001},"last":14.782}]},{"osd":5,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":15.275,"5min":19.556999999999999,"15min":21.327000000000002},"min":{"1min":11.198,"5min":10.254,"15min":10.254},"max":{"1min":19.477,"5min":97.191000000000003,"15min":97.191000000000003},"last":20.234000000000002},{"interface":"front","average":{"1min":16.013000000000002,"5min":18.779,"15min":20.585999999999999},"min":{"1min":10.664,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":19.811,"5min":64.048000000000002,"15min":64.048000000000002},"last":18.539000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:01:27 2023","interfaces":[{"interface":"back","average":{"1min":14.233000000000001,"5min":16.07,"15min":17.100000000000001},"min":{"1min":9.3729999999999993,"5min":8.7230000000000008,"15min":8.7230000000000008},"max":{"1min":18.184000000000001,"5min":74.064999999999998,"15min":74.064999999999998},"last":17.215},{"interface":"front","average":{"1min":16.649000000000001,"5min":20.157,"15min":21.806000000000001},"min":{"1min":10.581,"5min":10.055,"15min":10.055},"max":{"1min":20.405000000000001,"5min":92.477999999999994,"15min":92.477999999999994},"last":21.704000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:02:15.574 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-06T08:02:15.574 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:15.631 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:15.629+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:15.691 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:15.692 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:15.692 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:15.692 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:15.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:15.731 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:15.731+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:15.832 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:15.831+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:15.875 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:15.875 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:15.876 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:15.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:15.933 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:15.932+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:16.013 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:16.014 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:16.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:16.014 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:16.014 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:16.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:16.034 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:16.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:16.135 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:16.135+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:16.149 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:16.150 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:16.150 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:16.150 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:16.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:16.236 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:16.235+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:16.286 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:16.286 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:16.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:16.337 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:16.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:16.438 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:16.437+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:16.472 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 277.05131384399999, 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:16.473 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:16.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:16.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.478 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:16.478 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 96.887815079000006, 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:16.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:16.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:16.485 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 36.880754064999998, 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:16.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:16.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:16.489 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:16.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:16.538 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:16.537+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:16.620 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:16.621 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:16.621 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:16.621 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:16.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:16.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:16.637 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:02:16.676 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:16.675+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:16.755 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:16.749+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:16.762 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:16.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:16.762 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:16.763 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:16.763 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:16.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:16.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:16.875 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:16.875 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:16.873+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:16.942 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:16.941+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:16.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:16.976 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:16.976 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:16.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:16.976 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:16.976 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:16.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:17.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:17.043 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:17.043 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:17.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:17.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:17.127 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:17.145 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:17.144+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:17.160 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:17.160 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:17.160 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:17.160 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:17.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:17.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:17.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:17.218 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:17.245 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:17.243+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:17.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:17.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:17.346 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:17.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:17.354 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:17.354 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:17.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:17.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:17.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:17.446 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:17.445+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:17.495 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:17.495 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:17.495 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:17.496 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:17.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:17.548 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:17.547+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:17.632 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:17.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:17.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:17.648 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:17.647+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:17.754 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:17.753+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:17.773 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:17.773 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:17.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:17.867 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:17.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:17.874 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":43,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:17.950 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:17.951 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:17.951 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:17.951+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:17.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:18.052 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:18.052+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:18.129 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.129 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:18.182 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:18.182+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:18.254 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:18.254+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:18.269 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.269 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:18.363 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:18.362+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:18.449 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.457 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:18.456+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:18.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:18.558 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:18.556+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:18.590 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.590 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:18.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:02:18.685 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:18.684+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:18.760 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:18.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:18.798 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.798 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:18.893 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:18.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:18.962 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:18.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:18.976 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:18.976 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:18.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:19.071 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:19.069+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:19.156 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:19.156 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:19.164 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:19.163+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:19.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:19.265 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:19.264+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:19.336 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:19.336 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:19.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:19.386 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:19.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:19.467 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:19.466+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:19.507 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:02:19.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:19.533 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:19.533 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:19.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:19.568 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:19.568+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:19.669 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:19.668+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:19.731 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:19.731 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:19.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:19.770 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:19.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:19.784 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:19.784 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:19.871 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:19.869+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:19.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:19.878 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:19.944 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:19.945 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:19.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:19.995 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:19.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:20.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:20.003 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:20.093 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:20.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:20.093 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:20.092+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:20.126 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:20.126 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:20.126 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:20.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:20.181 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:20.179+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:20.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:20.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:20.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:20.281 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:20.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:20.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:20.309 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:20.310 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:20.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:20.371 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:20.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:20.375 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:20.374+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:20.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:20.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:20.476 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:20.476+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:20.504 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:20.504 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:20.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:20.575 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:20.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:20.575 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-12-06T08:02:20.575 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-12-06T08:02:20.575 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-12-06T08:02:20.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-12-06T08:02:20.597 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:20.597+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:20.678 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:20.678+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:20.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 281.26950060500002, 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:20.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:20.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.10600184, 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:20.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:20.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.098940826000003, 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:20.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:20.702 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:20.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:20.728 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:20.728 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2023-12-06T08:02:20.728 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:20.781 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:20.779+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:20.832 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:20.832 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:20.832 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:20.832 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:20.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:20.880 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:20.879+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:20.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:20.981 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:20.980+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:21.010 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.010 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:21.010 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:21.010 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:21.010 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:21.104 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:21.103+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:21.184 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:21.183+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:21.187 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.187 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:21.188 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:21.285 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:21.284+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:21.364 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:21.364 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:21.414 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:21.414+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:21.486 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:21.486+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:21.502 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":46,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:21.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:02:21.544 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:21.544 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:21.594 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:21.594+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:21.689 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:21.689+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:21.762 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.78148658200001, 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:02:21.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.117705581, 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.766 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:02:21.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:02:21.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.11442495199998, 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.80964833600001, 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:02:21.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:02:21.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.80880523899998, 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:21.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:02:21.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:02:21.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.781 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.78626973299998, 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:02:21.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:02:21.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.78427731400001, 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.783745049, 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:02:21.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:02:21.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:02:21.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.76682013999999, 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:02:21.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.765983506, 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:02:21.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.74153623799998, 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:21.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:02:21.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.73563387399997, 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:02:21.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.73092787799999, 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.72862600299999, 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:02:21.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:02:21.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.72812768, 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:21.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:02:21.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:02:21.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:02:21.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:02:21.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.72767376899998, 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:02:21.822 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:02:21.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:02:21.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:02:21.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.72645136900002, 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.826 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:02:21.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:02:21.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:02:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 293.08128538800003, 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:02:21.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:02:21.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 293.073846973, 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.834 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 293.06377360099998, 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:21.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:21.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:02:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:21.842 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:21.844 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:21.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:21.945 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:21.944+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:21.979 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:21.979 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:21.979 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:21.979 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:21.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:22.046 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:22.046+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:22.120 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:22.120 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:22.120 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:22.120 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:22.120 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:22.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:22.148 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:22.148+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:22.249 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:22.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:22.304 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:22.304 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:22.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:22.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:22.304 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:22.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:22.358 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:22.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:22.442 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:22.442 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:22.442 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:22.442 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:22.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:22.451 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:22.450+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:22.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:22.551 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:22.551+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:22.620 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:22.620 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:22.620 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:22.621 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:22.621 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:22.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:22.652 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:22.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:22.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:22.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:22.752 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:22.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:22.761 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.18171158799998, 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:22.762 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.06453424199999, 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:22.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:22.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:22.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.771 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 42.057730026000002, 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:22.772 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:22.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:22.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:22.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:22.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:22.779 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:22.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:22.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:22.816 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:22.853 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:22.853+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:22.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:22.904 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:22.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:22.954 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:22.954+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:22.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:22.990 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:23.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:23.074 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.073+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.094 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.095 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:23.145 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:02:23.159 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:23.160 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:23.159+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:23.160 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:23.257 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.256+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:23.276 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:23.289 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.290 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:23.357 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.381 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:23.381 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:23.438 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.438 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:23.459 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.458+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.560 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.575 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.576 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:23.662 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:23.661+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:23.714 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.715 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:23.758 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:17.712 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:23.762 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:23.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:23.809 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:17.763 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:23.863 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:23.863+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:23.895 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:23.895 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:23.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:23.964 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:23.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:24.032 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.032 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:24.053 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:18.007 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:24.085 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:24.084+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:24.166 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:24.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:24.276 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:24.276+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:24.365 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.365 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.368 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:24.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:24.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:24.469 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:24.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:24.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:24.564 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.564 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.570 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:24.569+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:24.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:24.670 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:24.670+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:24.758 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.759 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:24.771 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:24.770+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:24.871 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:24.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:24.896 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:24.896 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:24.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:24.972 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:24.972+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:25.033 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.033 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:25.084 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:25.084+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:25.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:02:25.174 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:25.173+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:25.210 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.210 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.218 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":50,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:25.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:25.304 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:25.303+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:25.376 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:25.375+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:25.389 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.389 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:25.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:25.484 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:25.484+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:25.571 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.571 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.578 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:25.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:25.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:25.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:25.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:25.679 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:25.679+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:25.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:25.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:25.738 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:25.738 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:25.738 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2023-12-06T08:02:25.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2023-12-06T08:02:25.747 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.747 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:25.784 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:25.784+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:25.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:25.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:25.886 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:25.886+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:25.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:25.934 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:25.934 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:25.934 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:25.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:25.986 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:25.985+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:26.043 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:26.044 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:26.083 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:26.083+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:26.085 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:26.085 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:26.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:26.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:26.151 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:26.184 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:26.184+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:26.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:26.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:26.284 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:26.285 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:26.285 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:26.285 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:26.285+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:26.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:26.346 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:26.346 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:26.352 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 18 (0%) 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:avg 2.25 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 0.968246 -> 0.968246 (expected baseline 1.40312) 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.0 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.1 with 3 -> 3 pgs (1.33333 -> 1.33333 * mean) 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0003 2023-12-06T08:02:26.380 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0030 2023-12-06T08:02:26.381 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2023-12-06T08:02:26.386 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:26.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:26.392 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2023-12-06T08:02:26.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2023-12-06T08:02:26.463 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:26.464 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:26.464 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:26.464 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:26.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:26.514 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:26.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:26.588 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:26.587+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:26.641 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:26.641 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:26.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:26.689 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:26.688+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 286.20338696300001, 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.788 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:26.788 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.08620961699999, 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.079405401000002, 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:26.796 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:26.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:26.798 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:26.797+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:26.898 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:26.898+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:26.928 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:26.928 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:26.928 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:26.928 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:26.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 18 (0%) 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:avg 2.25 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 0.968246 -> 0.968246 (expected baseline 1.40312) 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.0 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.1 with 3 -> 3 pgs (1.33333 -> 1.33333 * mean) 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2023-12-06T08:02:26.988 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2023-12-06T08:02:26.989 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0003 2023-12-06T08:02:26.989 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0030 2023-12-06T08:02:26.989 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2023-12-06T08:02:27.000 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:26.999+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:27.000 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2023-12-06T08:02:27.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2023-12-06T08:02:27.068 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:27.068 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:27.068 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:27.068 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:27.068 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:27.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:27.123 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:27.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:27.201 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:27.201+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:27.210 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:27.210 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:27.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:27.211 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:27.211 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:27.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:27.305 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:27.305+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:27.402 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:27.402+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:27.407 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:27.407 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:27.407 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:27.407 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:27.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:27.503 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:27.503+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:27.604 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:27.604+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:27.612 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:27.612 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:27.612 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:27.612 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:27.613 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:27.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 18 (0%) 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:avg 2.25 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 0.968246 -> 0.968246 (expected baseline 1.40312) 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.0 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.1 with 3 -> 3 pgs (1.33333 -> 1.33333 * mean) 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0003 2023-12-06T08:02:27.632 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0003 2023-12-06T08:02:27.633 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2023-12-06T08:02:27.645 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2023-12-06T08:02:27.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2023-12-06T08:02:27.705 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:27.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:27.806 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:27.806+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.835 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 307.69817607900001, 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:02:27.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:02:27.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 306.22706262600002, 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:02:27.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:02:27.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:02:27.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.996870247, 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.854 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.99616269799998, 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:02:27.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:02:27.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:02:27.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.862 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.99552570399999, 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:02:27.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:02:27.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:02:27.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:02:27.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:02:27.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.93846050899998, 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:02:27.869 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:02:27.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:02:27.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:02:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.874 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.93696973800002, 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.875 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.90462007600001, 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.881 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.886 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.897989301, 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.887 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:02:27.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:02:27.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 305.89733862000003, 2023-12-06T08:02:27.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.898 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.89386592900001, 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:27.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:02:27.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.89272720700001, 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.89177950999999, 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.909 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:02:27.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:02:27.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.86341551599998, 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.914 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.83520599899998, 2023-12-06T08:02:27.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:02:27.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:02:27.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.924 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.225008065, 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:02:27.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:02:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.22436322499999, 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:02:27.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:02:27.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:02:27.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.20389647100001, 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:02:27.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:02:27.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.20314755499999, 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.940 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:02:27.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:02:27.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.20250548600001, 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:27.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:02:27.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:02:27.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:27.950 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:27.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:27.952 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:27.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:28.053 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:28.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:28.131 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:28.132 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:28.132 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:28.132 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:28.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:28.184 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:28.184+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:28.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:28.246 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 18 (0%) 2023-12-06T08:02:28.246 INFO:teuthology.orchestra.run.smithi084.stdout:avg 2.25 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 0.968246 -> 0.968246 (expected baseline 1.40312) 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.0 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.1 with 3 -> 3 pgs (1.33333 -> 1.33333 * mean) 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0003 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0030 2023-12-06T08:02:28.247 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2023-12-06T08:02:28.255 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:28.255+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:28.259 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2023-12-06T08:02:28.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2023-12-06T08:02:28.296 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:28.296 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:28.296 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:28.296 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:28.296 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:28.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:28.356 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:28.355+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:28.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:28.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:02:28.457 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:28.456+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:28.502 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:28.502 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:28.503 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:28.503 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:28.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:28.557 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:28.557+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:28.646 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:28.646 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:28.646 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:28.646 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:28.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:28.697 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:28.696+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:28.759 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:28.759+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:28.850 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:28.850 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:28.860 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:28.860+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:28.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:28.961 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:28.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:29.032 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 289.60738366700002, 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:29.033 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:29.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:29.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 109.44388490199999, 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:29.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:29.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:29.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:29.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:29.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.436823887999999, 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:29.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:29.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:29.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:29.047 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:29.047 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:29.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:29.062 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:29.062+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:29.078 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":54,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:29.164 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:29.163+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:29.188 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:29.188 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:29.188 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:29.188 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:29.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:29.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:29.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 18 (0%) 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:avg 2.25 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 0.968246 -> 0.968246 (expected baseline 1.40312) 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.0 with 1 -> 1 pgs (0.444444 -> 0.444444 * mean) 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.1 with 3 -> 3 pgs (1.33333 -> 1.33333 * mean) 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0003 2023-12-06T08:02:29.264 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0030 2023-12-06T08:02:29.265 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2023-12-06T08:02:29.265 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:29.264+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:29.331 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:29.331 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:29.331 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:29.331 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:29.331 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:29.335 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:29.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:29.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:29.365 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:29.365+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:29.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:29.423 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:29.466 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:29.465+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:29.470 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:29.470 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:29.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:29.470 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:29.470 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:29.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:29.513 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:29.513 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:29.537 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:02:29.551 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:25.984 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.594 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:29.594+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:29.598 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:26.030 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.598 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:45.510 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.602 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:45.514 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:29.611 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:29.619 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:26.052 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.625 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:45.536 109213== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:29.637 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:29.638 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:29.638 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:29.638 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:29.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:29.693 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:29.692+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:29.698 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:29.698 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:29.778 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:29.778 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:29.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:29.789 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:29.789 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:29.787+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:29.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:29.870 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:29.870+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:29.910 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:29.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:29.924 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:29.924 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:29.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:29.972 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:29.972+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:30.072 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:30.072+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:30.098 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.098 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:30.173 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:30.173+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:30.236 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.236 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:30.275 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:30.274+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:30.374 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:30.374+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:30.418 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.418 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:30.516 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:30.515+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:30.576 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:30.576+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:30.606 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.606 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:30.646 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:24.597 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:30.690 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:24.599 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:30.691 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:24.621 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:30.692 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:30.691+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:30.778 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.778 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:30.793 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:30.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:30.893 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:30.893+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:30.919 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:30.919 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:30.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:30.994 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:30.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:31.059 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.059 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:31.095 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:31.094+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:31.196 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:31.196+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:31.240 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.240 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:31.296 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:31.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:31.399 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:31.399+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:31.430 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.430 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:31.500 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:31.500+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:31.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:02:31.576 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.577 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:31.600 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:31.600+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:31.701 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:31.701+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:31.786 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.786 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:31.836 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:31.836+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:31.903 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:31.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:31.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:31.965 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:31.965 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:31.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:32.018 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:32.018+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:32.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:32.105 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:32.105+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:32.196 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:32.196 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:32.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:32.209 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:32.209 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:32.209 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:32.208+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:32.304 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:32.305 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:32.307 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:32.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:32.353 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:32.353 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:32.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:32.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:32.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:32.412 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:32.412+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:32.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:32.505 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:32.508 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:32.508+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:32.534 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:32.534 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:32.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:32.534 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:32.534 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:32.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:32.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:32.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:32.609 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:32.609+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:32.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:32.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:32.710 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:32.710+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:32.720 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:32.720 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:32.720 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:32.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:32.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:32.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:32.812 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:32.811+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:32.818 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:02:32.897 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:32.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:32.912 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:32.912+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:32.933 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:32.933 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:32.934 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:32.934 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:32.934 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:32.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:32.951 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:32.963 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T08:02:32.964 INFO:tasks.mon_recovery:removing mon c 2023-12-06T08:02:32.964 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T08:02:32.964 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:02:33.014 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:33.013+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:33.018 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:02:33.008+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 109208) UID: 0 2023-12-06T08:02:33.019 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:02:33.009+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T08:02:33.096 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:33.096 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:33.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:33.096 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.67502800800003, 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:33.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:33.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:33.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:33.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:33.105 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 113.511529243, 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:33.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 53.504468228999997, 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:33.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:33.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:33.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:33.113 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:33.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:33.115 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:33.115+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:33.216 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:33.215+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:33.304 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:33.304 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:33.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:33.317 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:33.316+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:33.417 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:33.417+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:33.487 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:33.487 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:33.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:33.518 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:33.518+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:33.520 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:49.432 109213== 2023-12-06T08:02:33.520 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:49.432 109213== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:02:33.532 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:02:33.619 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:33.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:33.665 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:33.666 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:33.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:33.721 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:33.720+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:33.819 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:33.819 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:33.821 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:33.821+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:33.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:33.922 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:33.922+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:33.996 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:33.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:33.996 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:33.996 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:33.996 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:34.023 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:34.023+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:34.124 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:34.124+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 313.232590195, 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:02:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:02:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.56880919399998, 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.225 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.56552856500002, 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:02:34.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:02:34.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:02:34.230 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.260751949, 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:02:34.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:02:34.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.25990885200002, 2023-12-06T08:02:34.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.23737334600003, 2023-12-06T08:02:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:34.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:02:34.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.23538092699999, 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:34.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:02:34.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:02:34.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.23484866199999, 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:02:34.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:02:34.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.21792375299998, 2023-12-06T08:02:34.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:02:34.266 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 312.21708711899998, 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.19263985100002, 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:34.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:02:34.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:02:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.18673748700002, 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:02:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:02:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.18203149099998, 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.17972961599997, 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.17923129299999, 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.17877738200002, 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:02:34.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.177554982, 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:02:34.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 305.53238900100001, 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:34.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:02:34.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 305.52495058599999, 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 305.51487721400002, 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:34.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:34.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:02:34.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:34.312 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.313 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:34.313 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:34.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:34.315 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:34.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:34.416 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:34.416+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:34.483 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:34.484 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:34.484 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:34.484 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:34.517 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:34.517+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:34.618 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:34.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:34.620 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:34.620 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:34.620 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:34.620 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:34.620 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:34.718 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:34.718+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:34.759 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:34.759 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:34.760 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:34.760 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:34.819 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:34.819+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:34.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:02:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:34.896 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:34.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:34.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:34.957 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:34.955+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:35.021 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:35.021+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:35.063 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:35.063 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:35.063 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:35.063 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:35.064 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:35.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:35.124 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:35.123+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:35.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:35.151 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 294.63139717600001, 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:35.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:35.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 114.51421983, 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:35.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:35.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:35.221 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.507415614000003, 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:35.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:35.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:35.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:35.225 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:35.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:35.227 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:35.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:35.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:35.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:35.328 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:35.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:35.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:35.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:35.378 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:35.378 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:35.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:35.420 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:35.421 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:35.465 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:35.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:35.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:35.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:35.530 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:35.530+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:35.571 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:35.572 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:35.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:35.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:35.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:35.666 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:35.666+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:35.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:35.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:35.732 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:35.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:35.751 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:35.751 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:35.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:35.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:35.795 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:35.848 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:35.848+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:35.928 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:35.928 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:35.934 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:35.934+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:35.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:36.035 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:36.035+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:36.105 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.105 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:36.159 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:36.159+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:36.237 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:36.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:36.244 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.244 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:36.338 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:36.338+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:36.422 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.422 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:36.475 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:36.475+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:36.540 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:36.539+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:36.558 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.559 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:36.652 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:36.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:36.735 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.735 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.741 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:36.741+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:36.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:36.842 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:36.842+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:36.919 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:36.919 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:36.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:36.943 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:36.942+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:37.043 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:37.043+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:37.098 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.098 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:37.148 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:37.148+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:37.245 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:37.245+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:37.274 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.274 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:37.346 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:37.345+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:37.423 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.424 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:37.474 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:37.474+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:37.548 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:37.547+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:37.604 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.605 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:37.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T08:02:37.650 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:37.649+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:37.767 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:37.767+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:37.781 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.781 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:37.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:37.852 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:37.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:37.953 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:37.953+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:37.963 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:37.963 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:37.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:38.054 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:38.054+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:38.100 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.100 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:38.158 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:38.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:38.259 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:38.259+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:38.277 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.277 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:38.360 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:38.360+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:38.414 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.414 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:38.466 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:38.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:38.561 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:38.561+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:38.593 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.594 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:38.688 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:38.687+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:38.763 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:38.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:38.773 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.774 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:38.870 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:38.869+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:38.951 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:38.951 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:38.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:39.000 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:39.000+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:02:39.065 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T08:02:39.066 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T08:02:39.066 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2023-12-06T08:02:39.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:39.067 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:39.066+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:39.126 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.126 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:39.180 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:39.179+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:39.267 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.267 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.268 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:39.268+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:39.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:39.313 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:39.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:39.314 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:39.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2023-12-06T08:02:39.369 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:39.369+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:39.408 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.408 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:39.471 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:39.470+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:39.549 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.549 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:39.602 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:39.602+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:39.673 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:39.673+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:39.690 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.690 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:39.718 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:39.774 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:39.774+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:39.852 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.853 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:39.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:39.909 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:39.907+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:39.976 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:39.975+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:39.991 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:39.991 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:40.086 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:40.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:40.168 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:40.169 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:40.178 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:40.177+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:40.279 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:40.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:40.346 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:40.346 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:40.397 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:40.397+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:40.485 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:40.484+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:40.525 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:40.525 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:40.619 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:40.619+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:40.690 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:40.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:40.787 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:40.787+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:40.884 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:40.884 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:40.893 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:40.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:40.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:40.995 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:40.994+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:41.097 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:41.097+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:41.199 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:41.198+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:41.213 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.214 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:41.299 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:41.299+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:41.363 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.363 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:41.415 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:41.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:41.504 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:41.503+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:41.512 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.513 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:41.604 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:41.604+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:41.688 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.688 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:41.738 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:41.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:41.810 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:41.809+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:41.874 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:41.875 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:41.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:41.924 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:41.924+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:42.012 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:42.012+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:42.052 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.052 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:42.115 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:42.115+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:42.191 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.191 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:42.216 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:42.216+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:42.320 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:42.320+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:42.328 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.328 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:42.423 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:42.423+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:42.507 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.507 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:42.558 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:42.557+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:42.625 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:42.625+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:42.686 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.686 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:42.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:02:42.741 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:42.738+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:42.827 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:42.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:42.866 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:42.866 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:42.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:42.928 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:42.928+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:43.005 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.005 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:43.060 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:43.059+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:43.130 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:43.130+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:43.143 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.143 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:43.232 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:43.231+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:43.281 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.281 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:43.335 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:43.335+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:43.422 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.422 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:43.434 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:43.434+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:43.536 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:43.535+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:43.599 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.599 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:43.636 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:43.636+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:43.737 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:43.737+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:43.738 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.738 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:43.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T08:02:43.839 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:43.838+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:43.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:43.945 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:43.944+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:44.015 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.016 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:44.044 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:44.045+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:44.145 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:44.145+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:44.152 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.152 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:44.247 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:44.247+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:44.289 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.289 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:44.384 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:44.384+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:44.449 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:44.449+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:44.467 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.468 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:44.551 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:44.550+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:44.645 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.645 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.651 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:44.650+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:44.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:44.752 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:44.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:44.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:44.871 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:44.871+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:44.954 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:44.954+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:45.000 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.000 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:45.055 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:45.055+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:45.146 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.146 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:45.156 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:45.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:45.257 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:45.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:45.285 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.285 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:45.358 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:45.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:45.425 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.425 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:45.459 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:45.459+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:45.561 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:45.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:45.602 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.603 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:45.698 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:45.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:45.762 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:45.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:45.779 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.780 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:45.863 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:45.863+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:45.957 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:45.957 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:45.964 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:45.964+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:45.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:46.065 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:46.065+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:46.094 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.095 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:46.188 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:46.188+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:46.267 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:46.267+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:46.274 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.274 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:46.368 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:46.368+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:46.452 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.452 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:46.502 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:46.502+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:46.570 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:46.570+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:46.630 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.630 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:46.682 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:46.681+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:46.764 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.764 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.772 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:46.771+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:46.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:46.873 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:46.872+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:46.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:46.976 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:46.976+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:47.074 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:47.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:47.087 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.087 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:47.175 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:47.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:47.225 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.225 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:47.277 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:47.277+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:47.378 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:47.377+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:47.401 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.402 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:47.479 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:47.478+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:47.594 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:47.592+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:47.680 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:47.680+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:47.719 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.720 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:47.781 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:47.781+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:47.857 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:47.857 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:47.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:47.882 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:47.882+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:47.922 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:02:47.937 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2023-12-06T08:02:47.937 INFO:tasks.mon_recovery:causing some monitor log activity 2023-12-06T08:02:47.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2023-12-06T08:02:47.983 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:47.982+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:48.045 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:48.046 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:48.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:48.101 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:48.101+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:48.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:48.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:48.184 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:48.184+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:48.200 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:48.201 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:48.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:48.229 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:48.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:48.285 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:48.285+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:48.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:48.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:48.362 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:48.362 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:48.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:48.362 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:48.362 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:48.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:48.386 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:48.386+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:48.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:48.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:48.487 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:48.487+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:48.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:48.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:48.545 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:48.545 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:48.545 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:48.545 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:48.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:48.588 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:48.587+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:48.595 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:48.595 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:48.657 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:02:48.625+0000 ff5f640 -1 mon.a@0(leader).osd e85 definitely_dead 0 2023-12-06T08:02:48.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:48.688 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:48.688+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:48.700 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:48.701 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:48.701 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:48.701 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:48.701 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:48.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:48.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:48.789 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:48.789+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:48.892 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:48.892 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:48.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 309.46971214799999, 2023-12-06T08:02:48.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:48.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:48.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:02:48.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:02:48.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:48.900 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 129.306213383, 2023-12-06T08:02:48.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:02:48.901 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:48.901 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:48.901 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:02:48.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:48.906 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 69.299152368999998, 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:48.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:48.910 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:48.911 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:48.892+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:48.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:48.993 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:48.993+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:49.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2023-12-06T08:02:49.045 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:49.045 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:49.045 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:49.045 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:49.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:49.096 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:49.095+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:49.195 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:02:49.196 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:49.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:49.199 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.7. 2023-12-06T08:02:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:49.262 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:49.262 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:49.262 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:49.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:49.314 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:49.312+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:49.398 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:49.398+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:49.453 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:49.453 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:49.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:49.453 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:49.453 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:49.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:49.506 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:49.504+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:49.591 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:46.023 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.592 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:46.025 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.601 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:49.600+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:49.608 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:01:46.040 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.635 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:49.635 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:49.635 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:49.636 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:49.638 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:43.591 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.639 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:43.594 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:49.656 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:00:43.610 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:02:49.704 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:49.704+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:49.791 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:49.791 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:49.791 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:49.791 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:49.791 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:49.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:49.805 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:49.805+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:49.906 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:49.905+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:50.006 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:50.006+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:02:50.009 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 329.02889223400001, 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:02:50.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:02:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:02:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.36511123299999, 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:02:50.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:02:50.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.36183060399998, 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:02:50.018 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:02:50.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:02:50.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:02:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.05705398800001, 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:02:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:02:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:02:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.05621089099998, 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.027 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:02:50.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:50.031 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.03367538499998, 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:02:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.03168296600001, 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:02:50.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:02:50.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:02:50.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.031150701, 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:02:50.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:02:50.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.01422579199999, 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:02:50.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:02:50.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 328.013389158, 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:02:50.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:02:50.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:02:50.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.98894188999998, 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:50.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:02:50.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:02:50.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.056 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.98303952600003, 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.97833352999999, 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:50.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:02:50.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:02:50.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.97603165499999, 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:02:50.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:02:50.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:02:50.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.975533332, 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:02:50.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.97507942099998, 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:02:50.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.97385702100001, 2023-12-06T08:02:50.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:02:50.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:02:50.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:02:50.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.081 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 321.32869104000002, 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 321.321252625, 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.089 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 321.31117925299998, 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:50.094 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:50.094 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2023-12-06T08:02:50.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:50.137 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:50.137+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:50.220 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:50.216+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:50.235 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.235 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:50.235 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:50.235 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:50.332 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:50.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:50.410 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:50.410+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.415 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:50.415 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:50.415 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:50.511 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:50.510+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:50.595 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.595 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:50.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:50.595 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:50.595 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:50.647 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:50.645+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:50.713 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:50.712+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:50.775 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:50.775 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:50.775 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:50.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:50.829 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:50.829+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:50.914 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:50.913+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:50.954 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:50.954 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:50.954 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:50.954 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:50.954 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:50.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:51.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:51.021 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:51.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2023-12-06T08:02:51.048 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:51.048+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:51.118 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:51.115+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:51.139 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:51.139 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 310.55954275200003, 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:51.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:51.143 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 130.44236540599999, 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:51.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:51.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 70.435561190000001, 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:51.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:51.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:51.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:51.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:51.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:51.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.154 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:51.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:51.154 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:51.154 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:51.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:51.156 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:51.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:51.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:02:51.218 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:51.217+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:51.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:51.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:51.290 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:51.290 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:51.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:51.342 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:51.341+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:51.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:51.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:51.420 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:51.420+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:51.473 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:51.474 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:51.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:51.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:51.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:51.534 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:51.531+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:51.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:51.564 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:51.623 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:51.622+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:51.651 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:51.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:51.675 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:51.675 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:51.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:51.742 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:51.737+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:51.742 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:51.742 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:51.824 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:51.824+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:51.857 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:51.858 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:51.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:51.925 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:51.925+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:52.026 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:52.026+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:52.041 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.041 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:52.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2023-12-06T08:02:52.141 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:52.141+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:52.217 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:02:52.227 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:52.227+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:52.231 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.231 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:52.328 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:52.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:52.408 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.409 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:52.465 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:52.462+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:52.530 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:52.530+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:52.547 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.547 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:52.641 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:52.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:52.728 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.728 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.733 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:52.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:52.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:52.833 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:52.832+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:52.906 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:52.906 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:52.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:52.957 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:52.957+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:53.035 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:53.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:53.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2023-12-06T08:02:53.090 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.090 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:53.144 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:53.144+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:53.237 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:53.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:53.273 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.274 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:53.338 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:53.337+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:53.412 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.412 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:53.439 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:53.439+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:53.540 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:53.540+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:53.598 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.598 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:53.641 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:53.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:53.737 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.737 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.742 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:53.742+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:53.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:53.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:53.861 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:53.860+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:53.876 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:53.876 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:53.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:53.943 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:53.943+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:54.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:54.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:54.044 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:54.044+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:54.056 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:54.056 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:54.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:54.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:54.120 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:54.147 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:54.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:54.196 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:54.196 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:54.208 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:54.208 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:54.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:54.216 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:54.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:54.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:02:54.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2023-12-06T08:02:54.279 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:54.278+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:54.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:54.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:54.349 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:54.349+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:54.400 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:54.401 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:54.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:54.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:54.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:54.494 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:54.494+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:54.551 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:54.551+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:54.573 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:54.573 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:54.603 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:54.603 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:54.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:54.652 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:54.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:54.664 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:54.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:54.753 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:54.752+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:54.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:54.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:54.791 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:54.791 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:54.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:54.791 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:02:54.791 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:54.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:54.854 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:54.854+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:54.931 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:54.931 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:54.931 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:02:54.931 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:54.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:54.982 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:54.982+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:55.055 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:55.055+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:55.109 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:55.109 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:02:55.109 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:02:55.109 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:02:55.109 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:55.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:55.156 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:55.156+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:55.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2023-12-06T08:02:55.261 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:02:55.261 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:02:55.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 314.68130560600002, 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.262 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:02:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:02:55.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:55.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 134.56412825999999, 2023-12-06T08:02:55.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:02:55.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:55.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:02:55.270 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.557324043999998, 2023-12-06T08:02:55.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:02:55.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:55.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:02:55.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:02:55.276 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:02:55.276 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:55.261+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:55.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:55.357 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:55.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:55.426 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:55.427 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:55.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:55.477 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:55.477+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:55.559 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:55.559+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:55.605 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:55.605 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:55.606 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:55.606 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:55.606 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:55.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:02:55.660 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:55.660+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:55.734 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:02:55.755 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:55.755 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:55.761 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:55.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:55.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:02:55.862 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:55.862+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:55.945 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:55.945 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:55.945 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:55.945 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:55.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:02:55.997 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:55.997+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:56.063 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:56.063+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:56.123 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:56.123 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:56.123 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:56.123 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:56.123 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:56.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:02:56.173 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:56.173+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:56.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2023-12-06T08:02:56.265 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:56.266+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 336.23496596299998, 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.76385250999999, 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:02:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.53366013099998, 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.53295258200001, 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.53231558800002, 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:02:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.47525039300001, 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:02:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.47375962199999, 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:02:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.44140995999999, 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:02:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:02:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.43477918500002, 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:02:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:02:56.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:02:56.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:02:56.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.434128504, 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:02:56.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:02:56.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.43065581299999, 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:02:56.421 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:02:56.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:02:56.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.42951709099998, 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.433 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:56.434 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:56.434 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.42856939400002, 2023-12-06T08:02:56.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:02:56.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.434 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:56.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:02:56.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:02:56.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:02:56.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:02:56.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.4002054, 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:02:56.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:02:56.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.37199588300001, 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:02:56.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:02:56.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:02:56.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:02:56.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.76179794900003, 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:56.464 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.76115310900002, 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:02:56.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:02:56.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.74068635499998, 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.73993743900002, 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:02:56.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:02:56.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:02:56.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.73929536999998, 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:02:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:02:56.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:02:56.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:02:56.486 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:56.487 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:56.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:56.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:02:56.489 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:56.488+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:56.589 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:56.589+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:56.661 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:56.661 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:56.661 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:56.661 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:56.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:02:56.716 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:56.716+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:56.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:02:56.812 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:56.812+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:56.832 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:02:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:02:56.832 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:56.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:02:56.931 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:56.931+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:56.993 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:56.993+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:57.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:57.012 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:02:57.024 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:57.024 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:02:57.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:02:57.024 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:02:57.024 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:57.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:02:57.100 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:02:57.101 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:02:57.101 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:57.100+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:57.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:02:57.186 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:02:57.195 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:02:57.195+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:02:57.226 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:02:57.226 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:02:57.227 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:02:57.227 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:02:57.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:02:57.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2023-12-06T08:02:57.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:02:57.292 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:02:57.295 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:57.296+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:57.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:02:57.375 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:02:57.396 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:57.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:57.423 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:57.423 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:57.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:02:57.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:02:57.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:02:57.510 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:57.510+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:57.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:02:57.553 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:02:57.593 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:57.593 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:57.601 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:57.599+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:57.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:02:57.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:02:57.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:02:57.699 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:57.700+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:57.737 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:57.737 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:57.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:02:57.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:02:57.831 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:57.831+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:57.902 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:57.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:57.916 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:57.917 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:57.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:02:58.011 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:58.011+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:58.096 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.096 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.104 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:58.104+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:58.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:02:58.204 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:02:58.205+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:02:58.276 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.276 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:02:58.305 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:58.305+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:58.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2023-12-06T08:02:58.406 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:58.407+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:58.454 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.455 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:02:58.506 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:58.507+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:58.607 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:58.608+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:58.631 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.631 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:02:58.724 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:58.725+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:58.811 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.812 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.812 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:58.810+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:58.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:02:58.910 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:02:58.910+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:02:58.988 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:58.989 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:58.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:02:59.011 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:59.011+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:59.112 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:59.113+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:59.131 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:59.131 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:59.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:02:59.217 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:02:59.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:02:59.225 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:59.225+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:59.310 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:59.310 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:59.314 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:02:59.314+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:02:59.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:02:59.380 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:02:59.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2023-12-06T08:02:59.414 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:02:59.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:02:59.491 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:59.491 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:59.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:02:59.515 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:02:59.515+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:02:59.616 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:02:59.617+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:02:59.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:02:59.670 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:59.671 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:59.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:02:59.724 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:59.722+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:59.817 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:59.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:59.850 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:02:59.850 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:02:59.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:02:59.919 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:02:59.920+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:02:59.927 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:02:59.927 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:00.007 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:00.007 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:00.008 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:00.008 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:00.008 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:00.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:00.020 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:00.020+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:00.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:00.039 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:00.121 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:00.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:00.144 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:00.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:00.149 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:00.149 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:00.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:00.222 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:00.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:00.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:00.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:00.313 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:00.314 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:00.323 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:00.324+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:00.328 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:00.329 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:00.329 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:00.329 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:00.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:00.399 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:03:00.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:03:00.425 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:00.425+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:00.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2023-12-06T08:03:00.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:00.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:00.512 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:00.512 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:00.512 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:00.512 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:00.512 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:00.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:00.564 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:00.563+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:00.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:00.595 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:00.631 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:00.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:00.727 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:00.728+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:00.752 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:00.752 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:00.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:00.752 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 340.62187491399999, 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:00.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:03:00.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:03:00.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:03:00.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:00.762 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 339.150761461, 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:03:00.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:03:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:00.774 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.92056908199999, 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:03:00.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:03:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:03:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.781 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.91986153300002, 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:03:00.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.787 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.91922453900003, 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:00.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:03:00.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:03:00.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:03:00.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:03:00.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:03:00.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.796 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.86215934400002, 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.797 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:03:00.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.798 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:00.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:03:00.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:03:00.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:03:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.860668573, 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:03:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:03:00.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.828318911, 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:03:00.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:03:00.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:03:00.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:03:00.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:03:00.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:00.823 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.82168813599998, 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:03:00.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.829 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.82103745500001, 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:03:00.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.817564764, 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.837 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:03:00.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:03:00.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:03:00.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:03:00.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.81642604199999, 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:03:00.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:03:00.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:00.850 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.81547834499997, 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:03:00.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:03:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:00.856 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.78711435100001, 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:00.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:03:00.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:03:00.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:03:00.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.864 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.865 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:00.865 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:00.865 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.75890483400002, 2023-12-06T08:03:00.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:03:00.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:03:00.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.14870689999998, 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.872 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:03:00.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:00.877 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.14806205999997, 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:03:00.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:03:00.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:03:00.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:03:00.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.12759530599999, 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.883 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.12684639000003, 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:03:00.888 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.12620432099999, 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:00.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:03:00.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:03:00.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.898 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.899 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.899 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:00.899 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:00.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:00.901 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:00.901+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:01.002 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:01.003+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:01.032 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:01.033 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:01.103 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:01.104+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:01.173 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.173 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:01.174 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:01.174 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:01.174 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:01.207 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:01.207+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:01.307 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:01.307+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:01.310 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.310 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:01.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:01.310 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:01.310 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:01.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:03:01.407 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:01.408+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:01.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2023-12-06T08:03:01.504 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.505 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:01.505 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:01.505 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.508 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:01.508+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:01.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:01.611 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:01.611+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:01.688 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.688 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:01.688 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:01.688 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:01.688 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:01.710 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:01.711+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:01.811 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:01.812+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:01.843 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:01.843 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:01.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 322.422164431, 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:01.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:01.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:01.848 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 142.25866566600001, 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:01.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:01.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:01.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.852 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:01.853 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.853 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:01.854 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:01.854 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 82.251604651999997, 2023-12-06T08:03:01.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:01.854 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:01.854 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:01.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:01.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.858 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:01.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:01.859 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:01.859 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:01.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:01.913 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:01.912+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:01.989 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:01.989 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:01.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:02.013 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:02.013+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:02.115 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:02.114+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:02.137 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:02.137 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:02.137 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:02.137 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:02.137 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:02.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:02.215 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:02.215+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:02.274 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:02.274 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:02.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:02.275 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:02.275 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:02.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:02.327 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:02.328+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:02.424 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:02.424 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:02.424 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:02.424 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:02.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:03:02.425 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:02.424+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:02.524 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:02.525+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:02.598 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:02.599 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:02.599 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:02.599 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:02.599 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:02.600 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:03:02.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2023-12-06T08:03:02.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:03:02.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:02.658 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:02.657+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:02.725 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:02.726+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:02.821 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:03:02.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 341.83922474799999, 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:03:02.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:03:02.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:03:02.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:03:02.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 341.17544374699997, 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:03:02.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:03:02.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.839 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 341.17216311800001, 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:03:02.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:03:02.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:03:02.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:03:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:03:02.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.86738650199999, 2023-12-06T08:03:02.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.857 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.86654340500002, 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:02.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:03:02.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:03:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:02.864 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.84400789900002, 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.84201547999999, 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.869 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:03:02.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:03:02.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.84148321499998, 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.875 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.878 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:02.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.82455830599997, 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:03:02.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:03:02.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:03:02.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.82372167199998, 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.79927440400002, 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.898 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:03:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:03:02.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.79337204000001, 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.78866604400002, 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:03:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:03:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.78636416900002, 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:03:02.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:03:02.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:03:02.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.78586584599998, 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:03:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.78541193500001, 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:03:02.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:03:02.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:03:02.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 339.784189535, 2023-12-06T08:03:02.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:02.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:03:02.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:03:02.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 334.139023554, 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:02.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:03:02.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 334.13158513899998, 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:02.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:03:02.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 334.12151176700002, 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:02.953 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:03:02.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:02.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:02.957 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:02.958 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:02.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:02.959 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:02.958+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:02.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:02.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:02.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:03.059 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:03.059+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:03.082 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:03.082 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:03.130 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:03.130 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:03.130 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:03.131 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:03.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:03.169 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:03.169 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:03.185 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:03.185+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:03.258 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:03.258 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:03.261 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:03.261+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:03.276 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:03.277 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:03.277 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:03.277 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:03.277 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:03.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:03.355 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:03:03.355 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:03:03.372 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:03.373+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:03.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:03.450 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:03.463 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:03.463+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:03.489 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:03.489 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:03.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:03.553 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:03.553 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:03.565 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:03.564+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:03.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2023-12-06T08:03:03.671 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:03.671 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:03.675 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:03.675+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:03.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:03.766 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:03.767+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:03.818 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:03.818 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:03.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:03.867 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:03.867+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:03.968 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:03.969+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:03.995 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:03.995 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:04.071 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:04.071+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:04.135 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:04.136 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:04.189 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:04.189+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:04.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:03:04.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:03:04.219 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-06T08:03:04.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-06T08:03:04.275 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:04.274+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:04.277 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:04.277 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:04.377 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:04.376+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:04.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-06T08:03:04.462 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:04.462 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:04.514 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:04.514+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:04.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2023-12-06T08:03:04.582 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:04.581+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:04.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:03:04.648 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:04.648 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:04.682 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:04.683+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:04.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-06T08:03:04.828 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:04.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:04.842 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:04.842 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:04.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:04.891 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:04.891+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:04.994 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:04.993+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:05.025 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.026 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:05.094 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:05.093+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:05.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-06T08:03:05.190 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.190 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.194 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:05.194+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:05.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:05.300 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:05.295+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:05.385 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.385 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2023-12-06T08:03:05.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:05.397 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:05.396+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:05.497 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:05.497+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:05.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:03:05.571 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.571 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2023-12-06T08:03:05.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:05.629 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:05.628+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:05.702 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:05.701+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:05.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2023-12-06T08:03:05.775 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.775 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:05.826 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:05.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:05.828 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:05.827+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:05.907 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:05.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:05.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:05.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:05.974 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:05.974 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:05.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2023-12-06T08:03:05.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:06.004 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:06.005+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:06.032 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:06.032 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:06.106 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:06.106+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:06.159 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:06.159 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:06.202 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 326.781612129, 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 146.61811336400001, 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 86.611052349999994, 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:06.216 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:06.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:06.220 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:06.220+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:06.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:06.249 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:06.273 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:03:06.329 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:06.329+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:06.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:03:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:03:06.355 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:06.355 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:06.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:06.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:06.444 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:06.445 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:06.445+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:06.502 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:06.502 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:06.502 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:06.503 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:06.503 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:06.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:06.521 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:06.521+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:06.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:06.631 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:06.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:06.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2023-12-06T08:03:06.689 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:06.689 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:06.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:06.689 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:06.689 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:06.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:06.724 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:06.723+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:06.824 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:06.824+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:06.869 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:06.869 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:06.869 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:06.869 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:06.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:03:06.924 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:06.925+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:07.013 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:07.014 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:07.014 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:07.014 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:07.014 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:07.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:03:07.026 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:07.025+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:07.128 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:07.125+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.236 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:1 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'1, mlcod=0'0)", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.25425047300001, 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16386208299999999, 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941178+0000", 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.941363+0000", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185143 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.945685+0000", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043218739999999999 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.946623+0000", 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093768400000000004 2023-12-06T08:03:07.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.955964+0000", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093405229999999999 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.981102+0000", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025138773 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:20.990527+0000", 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094249639999999992 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.092319+0000", 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.10179215799999999 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.105040+0000", 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012720964 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.240 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.590469472, 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054281036999999997, 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604959+0000", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.604967+0000", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5889999999999997e-06 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607798+0000", 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028307430000000001 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.607909+0000", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111433 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608012+0000", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103068 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611289+0000", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032770439999999998 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.611670+0000", 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000380388 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.657842+0000", 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046172400000000002 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659240+0000", 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013983719999999999 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:22 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'12, mlcod=17'8)", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.58718884299998, 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051591867, 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608240+0000", 2023-12-06T08:03:07.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608252+0000", 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2524000000000001e-05 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608354+0000", 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101164 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608466+0000", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112176 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.608577+0000", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111279 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620564+0000", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011986533000000001 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.620978+0000", 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000414115 2023-12-06T08:03:07.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659302+0000", 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038324207999999998 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.659832+0000", 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052986799999999996 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:31 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'17, mlcod=17'14)", 2023-12-06T08:03:07.248 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.28241222700001, 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.073033785000000004, 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913017+0000", 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913028+0000", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1041e-05 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913337+0000", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030932799999999999 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913503+0000", 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016636099999999999 2023-12-06T08:03:07.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913625+0000", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121223 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.914773+0000", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011481600000000001 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.915151+0000", 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037788600000000002 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.974816+0000", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059665695999999997 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.986050+0000", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01123409 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:32 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=17'14)", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:07.252 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.28156912999998, 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093041278000000005, 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913860+0000", 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.913866+0000", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0249999999999999e-06 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.920859+0000", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006993193 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921003+0000", 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014432099999999999 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.921112+0000", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109059 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.925766+0000", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046533440000000002 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.926135+0000", 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036899600000000002 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.995346+0000", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069211769000000006 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006901+0000", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011554571 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:33 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=17'14)", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:07.256 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.25903362399998, 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.071053954000000002, 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936395+0000", 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936403+0000", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4780000000000002e-06 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936515+0000", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011273300000000001 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936642+0000", 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126885 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.936774+0000", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131734 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939509+0000", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027346440000000001 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.939889+0000", 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038032499999999999 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.006931+0000", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.067041817000000004 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007449+0000", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051833800000000002 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:34 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'14)", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.25704120500001, 2023-12-06T08:03:07.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.069566442000000006, 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938388+0000", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938394+0000", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1190000000000002e-06 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938504+0000", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010980300000000001 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938632+0000", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012867900000000001 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938729+0000", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6423999999999997e-05 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948044+0000", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093152019999999999 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.948458+0000", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041380900000000001 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007476+0000", 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.059018801000000003 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.007954+0000", 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047760499999999998 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'14)", 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.25650894, 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.072525791000000006, 2023-12-06T08:03:07.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938920+0000", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.938925+0000", 2023-12-06T08:03:07.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955135+0000", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016209286999999999 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955369+0000", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000234529 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955487+0000", 2023-12-06T08:03:07.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011779 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959091+0000", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00360415 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.959493+0000", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040138400000000002 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.010894+0000", 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.051401664999999999 2023-12-06T08:03:07.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011446+0000", 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055145499999999996 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:36 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'14)", 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.23958403099999, 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056124394000000001, 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.268 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955845+0000", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.955855+0000", 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0087e-05 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956033+0000", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017811899999999999 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956220+0000", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018747499999999999 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956411+0000", 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190731 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.964920+0000", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085085400000000002 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.965316+0000", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000396356 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011469+0000", 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046152499 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011969+0000", 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050058700000000004 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:37 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'14)", 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 345.238747397, 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055788872000000003, 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956681+0000", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956690+0000", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3939999999999996e-06 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.956863+0000", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017298300000000001 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957030+0000", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000166863 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.957180+0000", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015027899999999999 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968608+0000", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011428331 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:21.968984+0000", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037542400000000002 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.011989+0000", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043005038000000002 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.012470+0000", 2023-12-06T08:03:07.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048156000000000002 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:82 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'50)", 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.21430012899998, 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049085273999999998, 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981129+0000", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981142+0000", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295e-05 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981300+0000", 2023-12-06T08:03:07.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015851999999999999 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981476+0000", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017589799999999999 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.981607+0000", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013126500000000001 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.982957+0000", 2023-12-06T08:03:07.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013495479999999999 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.983343+0000", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038572999999999998 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.029647+0000", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046304658999999998 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.030214+0000", 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056670400000000001 2023-12-06T08:03:07.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:83 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'50)", 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.20839776499997, 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046827573999999997, 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:07.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987031+0000", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987040+0000", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9269999999999994e-06 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987165+0000", 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012546500000000001 2023-12-06T08:03:07.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.987889+0000", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072321899999999999 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988015+0000", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012663400000000001 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.988752+0000", 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073697599999999997 2023-12-06T08:03:07.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.989114+0000", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036143599999999997 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033137+0000", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044023281999999997 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033859+0000", 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072163499999999998 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.283 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:84 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'50)", 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.20369176899999, 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042654734, 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991737+0000", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.991745+0000", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5510000000000002e-06 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993532+0000", 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017875440000000001 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993671+0000", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138842 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.993771+0000", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9733000000000002e-05 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.997940+0000", 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041695029999999998 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.998302+0000", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036225500000000002 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.033881+0000", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035578679000000002 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034392+0000", 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051062700000000004 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:85 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'55, mlcod=18'50)", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.20138989399999, 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040866976999999999, 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994039+0000", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994045+0000", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880000000000001e-06 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994143+0000", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8160999999999997e-05 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994263+0000", 2023-12-06T08:03:07.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119856 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994361+0000", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8468000000000007e-05 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001466+0000", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071048350000000003 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.001818+0000", 2023-12-06T08:03:07.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035206300000000001 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034412+0000", 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032593833000000003 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034906+0000", 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049377299999999998 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.291 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:86 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'56, mlcod=18'50)", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.200891571, 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040936180000000003, 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994537+0000", 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994542+0000", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.186e-06 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994641+0000", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8336000000000002e-05 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994746+0000", 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010535999999999999 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994838+0000", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2138000000000005e-05 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004601+0000", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097624349999999999 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.004942+0000", 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089500000000002 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.034925+0000", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029983290999999999 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.035473+0000", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054853900000000001 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:87 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'57, mlcod=18'50)", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:07.295 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.20043765999998, 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043168064999999999, 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994991+0000", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.994996+0000", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1830000000000002e-06 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995859+0000", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086247099999999996 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.995966+0000", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010708 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996059+0000", 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3314999999999997e-05 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.007690+0000", 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011631139 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.008031+0000", 2023-12-06T08:03:07.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034034000000000001 2023-12-06T08:03:07.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.037396+0000", 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029365782999999999 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038159+0000", 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076275400000000004 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:88 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'58, mlcod=18'50)", 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:07.300 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 344.19921526000002, 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042493417999999998, 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996214+0000", 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996219+0000", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996316+0000", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7617999999999999e-05 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996422+0000", 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105268 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:22.996514+0000", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980999999999999e-05 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026325+0000", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029811612000000001 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.026770+0000", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044478900000000001 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038181+0000", 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011410652 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:23.038707+0000", 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052625800000000004 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:105 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'67, mlcod=18'64)", 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 338.55404927900003, 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048787200000000003, 2023-12-06T08:03:07.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641380+0000", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.641389+0000", 2023-12-06T08:03:07.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6739999999999992e-06 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648314+0000", 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069248790000000001 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648473+0000", 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015934099999999999 2023-12-06T08:03:07.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648582+0000", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010909 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651171+0000", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025881390000000002 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.651605+0000", 2023-12-06T08:03:07.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043466700000000002 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.688838+0000", 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037232596999999999 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.690167+0000", 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013288130000000001 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:106 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=18'64)", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 338.546610864, 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044031170000000001, 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648818+0000", 2023-12-06T08:03:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.648824+0000", 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9170000000000003e-06 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650094+0000", 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012701590000000001 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650205+0000", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011056399999999999 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.650331+0000", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012626099999999999 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.654866+0000", 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045351740000000003 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.655217+0000", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035107200000000002 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.691649+0000", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036432185999999998 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692849+0000", 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001199837 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.4170.0:107 1.0 e19/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=18'64)", 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 338.53653749199998, 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034586805999999998, 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:07.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658891+0000", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.658900+0000", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5150000000000006e-06 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659023+0000", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123397 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659810+0000", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078721399999999995 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.659925+0000", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114922 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662275+0000", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023500079999999998 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.662654+0000", 2023-12-06T08:03:07.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037829500000000001 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.692879+0000", 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030225484 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:28.693478+0000", 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059897100000000001 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:07.318 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:07.319 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:07.319 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:07.236+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:07.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:07.362 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:07.362+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:07.429 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:07.429+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:07.487 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:07.487 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:07.487 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:07.487 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:07.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:07.529 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:07.529+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:07.631 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:07.631+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:07.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2023-12-06T08:03:07.671 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:07.671 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:07.671 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:07.671 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:07.671 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:07.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:07.762 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:07.763+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:07.833 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:07.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:07.852 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:07.852 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:07.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:07.852 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:07.853 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:07.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:07.933 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:07.932+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:08.031 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:08.031 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:08.031 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:08.031 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:08.034 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:08.034+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:08.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:08.135 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:08.136+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:08.171 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:08.171 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:08.171 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:08.171 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:08.171 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:08.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:08.236 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:08.235+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:08.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:03:08.337 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:08.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 327.77652975400002, 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:03:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:03:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:03:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 147.65935240799999, 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 87.652548191999998, 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:03:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:03:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:08.375 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:08.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:08.438 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:08.437+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:08.506 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:08.507 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:08.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:08.539 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:08.540+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:08.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:08.640 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:08.640+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:08.695 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:08.695 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:08.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:08.745 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:08.742+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:08.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2023-12-06T08:03:08.845 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:08.843+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:08.890 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:08.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:08.891 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:08.891 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:08.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:08.981 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:08.982 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:08.986 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:08.984+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:09.047 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:09.046+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:09.090 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:09.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:09.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:09.148 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:09.148+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:09.177 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:09.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:09.262 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:09.263 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:09.263 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:09.263+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:09.307 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:09.307 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:09.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:09.346 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2023-12-06T08:03:09.346 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2023-12-06T08:03:09.351 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:09.351+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:09.431 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:09.431 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:09.451 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:09.450+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:09.490 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:09.491 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:09.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:09.531 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:09.532 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:09.551 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:09.552+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:09.652 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:09.652+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:09.660 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:09.660 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:09.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:09.754 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:09.754+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:09.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2023-12-06T08:03:09.803 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:09.803 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:09.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:09.854 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:09.855+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:09.906 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:03:09.955 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:09.955+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:10.000 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.001 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:10.057 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:10.056+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:10.142 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.142 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:10.193 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:10.192+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:10.258 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:10.257+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:10.322 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.323 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:10.375 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:10.372+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:10.461 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:10.461+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:10.502 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.503 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:10.561 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:10.561+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:10.639 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.639 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:10.689 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:10.689+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:10.762 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:10.762+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:10.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2023-12-06T08:03:10.825 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:10.825 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:10.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:10.876 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:10.876+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:10.966 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:10.966+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:11.007 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.007 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:11.067 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:11.067+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:11.168 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:11.168+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:11.186 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.186 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-12-06T08:03:11.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:03:11.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:03:11.252 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2023-12-06T08:03:11.253 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-06T08:03:11.253 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:03:11.269 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:11.268+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:11.287 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:11.280+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86701) UID: 0 2023-12-06T08:03:11.290 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:11.280+0000 b47e640 -1 osd.5 93 *** Got signal Terminated *** 2023-12-06T08:03:11.362 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.362 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-12-06T08:03:11.371 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:11.370+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:11.472 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:11.472+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:11.542 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.542 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:03:11.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:11.588 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:11.587+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:11.674 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:11.673+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:11.733 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.733 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:11.780 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:11.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:11.788 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:11.787+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:11.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2023-12-06T08:03:11.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:11.868 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:11.876 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:11.876+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:11.915 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:11.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:11.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:03:11.992 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:11.992 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:12.008 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:12.008+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:12.082 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:12.081+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:12.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:12.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:12.115 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:12.115 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:12.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:12.180 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:12.179+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:12.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:12.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:12.266 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:12.267 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:12.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:12.281 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:12.281+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:12.383 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:12.382+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:12.447 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:12.447 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:12.447 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:12.447 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:12.447 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:12.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:12.500 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:12.499+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:12.584 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:12.584+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:12.604 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:12.604 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:12.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:12.604 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:12.604 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:12.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:12.698 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:12.697+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:12.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2023-12-06T08:03:12.792 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:12.792 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:12.793 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:12.793 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:12.793 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:12.792+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:12.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:12.887 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:12.886+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:12.988 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:12.987+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:12.999 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:12.999 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:13.007 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:03:13.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:13.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:13.100 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:13.099+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:13.191 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:13.190+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:13.255 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:13.255 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:13.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:13.255 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.11113626100001, 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:03:13.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:03:13.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:03:13.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:03:13.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.64002280800003, 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:03:13.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:13.267 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.40983042900001, 2023-12-06T08:03:13.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:03:13.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.268 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:13.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:03:13.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.40912287999998, 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.276 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:03:13.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:03:13.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:03:13.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:03:13.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.40848588599999, 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:03:13.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:03:13.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:03:13.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.35142069099999, 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.289 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:13.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:03:13.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:03:13.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:03:13.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:03:13.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:03:13.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:03:13.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.300 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.34992992000002, 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:03:13.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:03:13.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:03:13.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.31758025800002, 2023-12-06T08:03:13.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:13.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:03:13.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:03:13.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.310949483, 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:03:13.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:03:13.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:13.329 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.31029880199998, 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:03:13.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:03:13.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:03:13.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:13.342 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.30682611100002, 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:03:13.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:03:13.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:03:13.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.30568738900001, 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:03:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.304739692, 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:03:13.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.27637569799998, 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.362 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:03:13.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:03:13.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:03:13.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.24816618099999, 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:13.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:03:13.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:03:13.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.637968247, 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:03:13.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:03:13.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:03:13.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:03:13.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:03:13.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:03:13.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.637323407, 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:03:13.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.61685665300001, 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:03:13.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.390 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:03:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.61610773699999, 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:13.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:03:13.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:03:13.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:13.407 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.61546566800001, 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:13.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:13.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:03:13.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:03:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:03:13.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:03:13.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:13.415 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:13.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:13.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:13.417 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:13.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:13.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:13.419 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:13.419 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:13.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:13.518 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:13.517+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:13.587 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:13.588 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:13.588 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:13.588 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:13.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:13.640 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:13.639+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:13.719 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:13.718+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:13.727 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:13.728 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:13.728 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:13.728 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:13.728 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:13.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:13.824 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:13.822+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:13.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2023-12-06T08:03:13.917 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:13.917 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:13.921 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:13.920+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:13.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:14.025 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:14.024+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:14.101 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:14.101 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:14.102 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:14.102 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:14.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:14.123 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:14.122+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:14.224 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:14.223+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:14.282 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:14.283 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:14.283 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:14.283 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:14.283 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:14.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:14.336 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:14.335+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:14.428 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:14.428 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:14.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:14.428 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:14.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 335.00727166000001, 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:14.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:14.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.843772895, 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:14.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:14.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:14.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:14.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:14.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.836711880999999, 2023-12-06T08:03:14.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:14.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:14.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:14.441 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:14.442 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:14.428+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:14.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:14.530 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:14.530+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:14.581 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:14.582 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:14.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:14.631 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:14.630+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:14.732 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:14.732+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:14.740 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:14.741 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:14.741 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:14.741 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:14.741 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:14.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:14.779 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:11.211 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.780 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:11.213 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.807 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:11.239 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.823 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:08.777 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.826 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:08.780 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.833 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:14.833+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:14.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-06T08:03:14.857 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:08.809 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:14.882 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:14.883 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:14.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:14.883 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:14.883 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:14.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:14.935 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:14.934+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:15.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:03:15.025 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:15.025 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:15.025 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:15.066 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:15.066+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:15.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2023-12-06T08:03:15.156 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:15.155+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:15.172 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.172 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:15.172 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:15.172 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:15.172 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:15.238 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:15.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:15.339 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:15.339+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:15.353 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.353 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:15.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:15.354 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:15.354 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:15.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:15.464 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:15.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:15.491 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.491 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:15.491 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:15.491 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:15.547 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:15.545+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:15.634 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.634 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:15.635 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:15.635 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:15.635 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.642 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:15.642+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:15.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:15.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:15.666 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:15.743 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:15.743+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:15.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:15.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:15.821 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:15.821 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:15.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:15.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:15.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 335.24087911300001, 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:03:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 155.123701767, 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:03:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:03:15.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.116897550999994, 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:15.833 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:15.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:03:15.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:15.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:15.837 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:15.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:15.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:15.904 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:15.905 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:15.904+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:15.945 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:15.944+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:15.982 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:15.982 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:15.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:16.001 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:16.002 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:16.046 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:16.045+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:16.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:16.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:16.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2023-12-06T08:03:16.105 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:16.121 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:03:16.147 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:16.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:16.147 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:16.147 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:16.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:16.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:16.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:16.249 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:16.248+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:16.295 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:16.296 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:16.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:16.335 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:16.335 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:16.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:16.350 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:16.348+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:16.449 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:16.449+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:16.516 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:16.517 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:16.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:16.570 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:16.570+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:16.651 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:16.651+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:16.666 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:16.666 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:16.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:16.752 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:16.751+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:16.847 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:16.847 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:16.853 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:16.852+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:16.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:16.954 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:16.953+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:17.023 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.023 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:17.074 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:17.074+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:17.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2023-12-06T08:03:17.166 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:17.166+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:17.221 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.221 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:17.273 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:17.273+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:17.355 INFO:tasks.ceph.osd.5:Stopped 2023-12-06T08:03:17.355 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2023-12-06T08:03:17.355 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:03:17.355 DEBUG:teuthology.orchestra.run.smithi107:> type ceph-objectstore-tool 2023-12-06T08:03:17.357 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:17.357+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:17.370 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-06T08:03:17.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2023-12-06T08:03:17.402 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.402 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:17.458 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:17.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:17.579 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:17.578+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:17.592 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.592 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:17.688 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:17.686+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:17.762 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:17.761+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:17.772 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.772 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:17.866 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:17.866+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:17.951 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:17.951 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:17.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:17.963 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:17.963+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:18.064 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:18.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:18.094 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.094 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:18.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:03:18.182 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:18.182+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:18.232 INFO:tasks.mon_recovery:adding mon c back in 2023-12-06T08:03:18.232 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-06T08:03:18.233 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-06T08:03:18.234 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.234 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:18.275 INFO:tasks.ceph.mon.c:Started 2023-12-06T08:03:18.275 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2023-12-06T08:03:18.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:03:18.289 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:18.288+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:18.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:03:18.367 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:18.367+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:18.371 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.371 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:18.468 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:18.468+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:18.510 INFO:teuthology.orchestra.run.smithi107.stdout:2.1 2023-12-06T08:03:18.510 INFO:teuthology.orchestra.run.smithi107.stdout:2.4 2023-12-06T08:03:18.511 INFO:teuthology.orchestra.run.smithi107.stdout:1.0 2023-12-06T08:03:18.553 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.553 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:18.570 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:18.570+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:18.671 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:18.671+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:18.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:18.733 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:18.734 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.734 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:18.789 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:18.788+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:18.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:18.842 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:18.873 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:18.873+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:18.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:18.920 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:03:18.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:18.967 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:18.995 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:18.995+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:19.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 2.4 --file /var/log/ceph/exp.2.4.5 2023-12-06T08:03:19.053 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:19.053 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:19.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:19.054 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:19.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:19.104 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:19.104+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:19.143 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:19.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:19.146 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:19.177 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:19.177+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:19.233 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:19.234 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:19.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:19.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:19.239 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:03:19.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:19.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:19.326 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:19.326 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:19.325+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:19.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:19.379 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:19.378+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:19.385 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 338.80440946300001, 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:19.386 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:19.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 158.68723211700001, 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.393 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 98.680427901000002, 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:03:19.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:19.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:19.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:19.402 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:19.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:19.479 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:19.479+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:19.536 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:19.536 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:19.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:19.589 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:19.588+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:19.681 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:19.681+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:19.722 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:19.722 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:19.722 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:19.722 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:19.722 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:19.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:19.781 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:19.781+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:19.883 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:19.882+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:19.921 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:19.921 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:19.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:19.921 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:19.921 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:19.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:19.985 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:19.984+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:20.081 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:20.081 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:20.081 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:20.081 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:20.088 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:20.086+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:20.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:Exporting 2.4 info 2.4( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2023-12-06T08:03:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:Export successful 2023-12-06T08:03:20.188 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:20.187+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:20.266 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:20.266 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:20.266 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:20.266 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:20.267 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:20.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:20.288 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:20.288+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:20.390 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:20.390+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:20.503 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:20.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 360.37264325899997, 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:03:20.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:03:20.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:03:20.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.90152980599999, 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:03:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.67133742700003, 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.670629878, 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:03:20.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.66999288400001, 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.612927689, 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:03:20.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:03:20.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:03:20.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:03:20.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:03:20.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.61143691799998, 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:03:20.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.546 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.57908725599998, 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:03:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:03:20.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:03:20.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:03:20.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.57245648100002, 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:03:20.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:03:20.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:03:20.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:03:20.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.562 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 358.57180579999999, 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.563 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:20.568 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.56833310899998, 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:03:20.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:03:20.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:03:20.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:03:20.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.56719438699997, 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:03:20.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:03:20.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:20.580 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.56624669000001, 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:20.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:03:20.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.537882696, 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:03:20.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:03:20.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:03:20.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:20.594 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.509673179, 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:03:20.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:03:20.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:03:20.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:03:20.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.89947524500002, 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.601 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:03:20.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.606 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.89883040500001, 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.607 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:20.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:03:20.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:03:20.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:03:20.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.87836365099997, 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.87761473500001, 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.619 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:03:20.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.87697266599997, 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:03:20.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:20.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:03:20.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:03:20.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:20.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:03:20.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:03:20.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:03:20.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:20.631 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:20.631 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:20.503+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:20.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:20.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 2.4 2023-12-06T08:03:20.673 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:20.673+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:20.734 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:20.734+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:20.804 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:20.804 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:20.804 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:20.804 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:20.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:20.855 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:20.854+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:20.936 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:20.936+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:20.988 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:20.988 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:20.988 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:20.988 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:20.988 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:20.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:21.042 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:21.042+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:21.133 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:21.133 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:21.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:21.133 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:21.133 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:21.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-06T08:03:21.138 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:21.137+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:21.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:21.238 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:21.238+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:21.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-12-06T08:03:21.315 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:21.315 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:21.315 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:21.315 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:21.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:21.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:21.371 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:21.370+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:21.447 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:21.447+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:21.461 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:21.461 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:21.461 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:21.462 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:21.462 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:21.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:21.559 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:21.559+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:21.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:21.610 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:21.643 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:21.641+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.22992565099997, 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:21.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 162.06642688599999, 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:21.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 102.059365872, 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:21.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:21.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:21.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:21.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:21.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:21.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:21.669 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:21.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:21.715 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:21.715 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:21.743 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:21.743+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: marking collection for removal 2023-12-06T08:03:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:setting '_remove' omap key 2023-12-06T08:03:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:finish_remove_pgs 2.4_head removing 2.4 2023-12-06T08:03:21.777 INFO:teuthology.orchestra.run.smithi107.stdout:Remove successful 2023-12-06T08:03:21.805 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:21.805 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:21.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:21.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:21.816 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:21.862 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:21.858+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:21.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:03:21.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:21.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:21.962 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:21.960+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:21.970 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:21.970 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:21.970 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:21.970 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:21.970 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:21.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:22.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:22.039 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:22.040 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:22.068 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:22.068+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:22.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:22.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:22.154 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:22.154 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:22.162 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:22.162+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:22.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:22.221 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:22.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:22.226 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:22.263 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:22.262+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:22.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.4.5 2023-12-06T08:03:22.341 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:22.341 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:22.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:22.363 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:22.363+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:22.464 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:22.464+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:22.537 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:22.537 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:22.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:22.566 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:22.565+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:22.616 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2023-12-06T08:03:22.649 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:03:22.666 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:22.666+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:22.740 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:22.740 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:22.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:22.794 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:22.793+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:22.869 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:22.868+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:22.920 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:22.920 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:22.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:22.969 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:22.969+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:23.061 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.061 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:23.074 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:23.070+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:23.187 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:23.186+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:23.200 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.201 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:23.272 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:23.271+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:23.339 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.339 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:23.373 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:23.372+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:23.474 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:23.474+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:23.519 INFO:teuthology.orchestra.run.smithi107.stdout:get_pg_num_history pg_num_history pg_num_history(e93 pg_nums {1={16=1},2={19=8}} deleted_pools ) 2023-12-06T08:03:23.519 INFO:teuthology.orchestra.run.smithi107.stdout:Importing pgid 2.4 2023-12-06T08:03:23.519 INFO:teuthology.orchestra.run.smithi107.stdout:write_pg epoch 61 info 2.4( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2023-12-06T08:03:23.520 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.520 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.521 INFO:teuthology.orchestra.run.smithi107.stdout:Import successful 2023-12-06T08:03:23.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:23.615 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:23.615+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:23.687 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:23.687+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:23.701 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.701 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:23.779 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:23.777+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:23.878 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:23.878+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:23.881 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:23.882 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:23.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:23.979 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:23.979+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:24.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /var/log/ceph/exp.2.4.5 2023-12-06T08:03:24.062 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:24.062 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:24.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:24.114 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:24.114+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:24.182 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:24.181+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:24.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:03:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:24.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:24.282 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:24.282+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:24.383 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:24.383+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:24.440 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:24.441 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:24.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:24.482 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:24.483 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:24.488 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:24.486+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:24.571 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:24.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:24.586 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:24.586+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:24.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-12-06T08:03:24.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:24.669 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:24.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:24.687 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:24.687+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:24.759 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:24.759 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:24.789 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:24.788+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:24.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:24.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:24.864 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:21.296 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:24.869 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:21.302 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:24.875 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:24.876 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:24.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:24.889 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:21.322 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:24.890 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:24.890+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:24.990 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:24.990+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:25.042 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:25.042 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:25.042 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:25.042 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:25.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:25.097 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:25.095+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:25.113 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:19.066 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:25.114 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:19.068 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:25.128 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:19.082 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:25.230 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:25.230+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:25.248 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:25.261 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:25.262 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:25.262 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:25.262 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:25.262 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:25.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:25.294 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:25.293+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:25.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:25.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:25.396 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:25.395+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:25.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:25.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 346.02872018199997, 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:25.450 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:25.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:25.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.86522141699999, 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:25.454 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:25.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:25.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.858160403, 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:25.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:25.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:25.464 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:25.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:25.507 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:25.507+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:25.593 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:25.593 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:25.593 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:25.593 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:25.597 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:25.597+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:25.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:25.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:03:25.709 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:25.708+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:25.730 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:25.730 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:25.730 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:25.730 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:25.730 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:25.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:25.803 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:25.802+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:25.873 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:25.873 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:25.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:25.873 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:25.873 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:25.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:25.903 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:25.903+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:26.007 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:26.006+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:26.054 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.054 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:26.054 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:26.054 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:26.108 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:26.108+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:26.211 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:26.211+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:26.237 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.237 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:26.237 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:26.238 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:26.238 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:26.275 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2023-12-06T08:03:26.336 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:26.336+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:26.354 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":38,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:03:26.388 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.388 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:26.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:26.389 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:26.389 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:26.415 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:26.415+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:26.516 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:26.516+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:26.531 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:26.532 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:26.549 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.213 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:26.617 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:26.616+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:26.669 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:26.669 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:26.722 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:26.720+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:26.773 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.437 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:26.818 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:26.818+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:26.851 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:26.851 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.27165172700001, 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:03:26.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.855 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 166.154474381, 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:26.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:03:26.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:03:26.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.14767016499999, 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:26.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:03:26.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:03:26.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:26.864 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:26.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:26.919 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:26.919+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:26.996 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:26.996 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:26.996 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:26.996 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:27.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:27.019 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:27.019+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:27.121 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:27.121+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:27.187 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:27.187 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:27.187 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:27.187 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:27.188 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:27.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:27.237 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:27.237+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:27.328 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:27.327+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:27.369 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:27.369 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:27.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:27.370 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:27.370 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:27.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:27.433 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:27.432+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:27.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:27.523 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:27.523 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:27.523 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:27.523 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:27.525 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:27.525+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:27.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:27.627 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:27.626+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:27.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:27.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:27.723 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:27.724 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:27.724 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:27.724 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:27.724 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:27.726 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:27.726+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:27.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:27.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:27.790 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:27.827 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:27.826+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:27.912 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:27.913 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:27.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:27.928 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:27.928+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:27.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:28.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:28.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:28.039 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:28.038+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:28.062 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.062 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:28.131 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:28.131+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:28.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:28.141 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:28.143 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:28.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:28.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:28.232 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:28.232+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:28.241 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.241 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:28.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:03:28.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:28.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:28.333 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:28.332+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:28.429 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.430 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.433 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:28.433+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:28.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:28.556 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:28.556+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:28.570 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.570 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:28.636 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:28.635+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:28.737 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:28.737+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:28.754 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.754 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:28.838 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:28.838+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:28.910 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:28.910 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:28.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:28.962 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:28.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:29.040 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:29.040+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:29.049 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.049 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:29.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2023-12-06T08:03:29.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-12-06T08:03:29.103 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-12-06T08:03:29.103 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.249436 2023-12-06T08:03:29.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.2494363548616415 2023-12-06T08:03:29.147 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:29.147+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:29.188 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.188 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:29.243 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:29.242+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:29.328 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.328 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:29.347 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:29.347+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:29.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:03:29.448 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:29.448+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:29.510 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.510 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:29.565 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:29.564+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:29.649 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.649 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.649 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:03:29.649+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:03:29.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:29.750 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:29.750+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:29.830 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.830 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.831 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:03:29.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:29.885 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:29.884+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:29.952 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:29.952+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:29.975 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:29.975 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:29.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:30.053 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:30.053+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:30.055 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":42,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:03:30.154 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:30.154+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:30.162 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:30.162 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:30.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:30.255 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:30.255+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:30.316 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:30.316 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:30.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-06T08:03:30.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-12-06T08:03:30.363 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.025 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:30.363 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:30.362+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:30.416 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.080 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:30.457 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:30.457+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:30.519 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2023-12-06T08:03:30.519 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-12-06T08:03:30.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-12-06T08:03:30.559 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:30.558+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:30.591 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:30.591 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:30.593 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 0.249436 (8163472) 2023-12-06T08:03:30.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:30.708 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:30.707+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:30.708 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:30.713 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:12.377 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:30.760 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:30.760+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:30.769 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:30.770 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:30.770 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:30.770 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:30.770 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:30.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-12-06T08:03:30.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:30.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:30.861 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:30.861+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:30.919 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:30.953 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:30.954 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:30.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:30.954 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:30.954 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:30.963 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:30.962+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:30.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-12-06T08:03:31.007 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:31.007 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:31.064 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:31.064+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:31.151 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:31.151 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:31.151 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:31.151 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:31.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-12-06T08:03:31.206 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:31.206+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:31.265 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:31.265+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:31.300 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:31.300 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:31.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-12-06T08:03:31.395 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:31.395+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:31.467 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:31.467+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:31.542 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:31.558 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 371.412883776, 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.070393119, 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.075441+0000", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.076109+0000", 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066786099999999995 2023-12-06T08:03:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.080930+0000", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048208510000000001 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.085402+0000", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044723940000000002 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.094048+0000", 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086460970000000002 2023-12-06T08:03:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.106629+0000", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012580653000000001 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.112657+0000", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060283400000000001 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.599282+0000", 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48662538999999999 2023-12-06T08:03:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.696209+0000", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096926346999999996 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.786301+0000", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090092346000000004 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:20.924739+0000", 2023-12-06T08:03:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13843786599999999 2023-12-06T08:03:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.118791+0000", 2023-12-06T08:03:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19405209200000001 2023-12-06T08:03:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.130524+0000", 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011732797 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.145834+0000", 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015310084999999999 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.94177032300001, 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12201603799999999, 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 21 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546554+0000", 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.546564+0000", 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3789999999999995e-06 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570240+0000", 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023676276 2023-12-06T08:03:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570380+0000", 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013998299999999999 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.570520+0000", 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013988499999999999 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.577104+0000", 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065841220000000004 2023-12-06T08:03:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.578656+0000", 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015519170000000001 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.583626+0000", 2023-12-06T08:03:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049696159999999996 2023-12-06T08:03:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.611834+0000", 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028208509999999999 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.666821+0000", 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054986814000000002 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.667829+0000", 2023-12-06T08:03:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001008192 2023-12-06T08:03:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.668570+0000", 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074134400000000001 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.711577944, 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16061942900000001, 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 30 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776747+0000", 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776757+0000", 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0171e-05 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.776921+0000", 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163898 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777073+0000", 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015210399999999999 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777216+0000", 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014278000000000001 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778758+0000", 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015426019999999999 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.780315+0000", 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015566110000000001 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.782728+0000", 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002413157 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877187+0000", 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094458643999999994 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.935613+0000", 2023-12-06T08:03:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058426354999999999 2023-12-06T08:03:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.936645+0000", 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001032437 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.937366+0000", 2023-12-06T08:03:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072066999999999997 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.71087039499997, 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22767222100000001, 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 31 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777454+0000", 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777464+0000", 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3370000000000006e-06 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777588+0000", 2023-12-06T08:03:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447099999999999 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777722+0000", 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133817 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.777866+0000", 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014383399999999999 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.786586+0000", 2023-12-06T08:03:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087198029999999999 2023-12-06T08:03:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.788200+0000", 2023-12-06T08:03:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016148650000000001 2023-12-06T08:03:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.790710+0000", 2023-12-06T08:03:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025093820000000001 2023-12-06T08:03:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877303+0000", 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086593587 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.003353+0000", 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12604929300000001 2023-12-06T08:03:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.004381+0000", 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001028812 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.005126+0000", 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074501999999999999 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.71023340099998, 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23502557700000001, 2023-12-06T08:03:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 32 2023-12-06T08:03:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778091+0000", 2023-12-06T08:03:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.778100+0000", 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1449999999999997e-06 2023-12-06T08:03:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802259+0000", 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024158775 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802397+0000", 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013789499999999999 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.802537+0000", 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013944200000000001 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.809894+0000", 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073574729999999998 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.812828+0000", 2023-12-06T08:03:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029338210000000001 2023-12-06T08:03:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.818305+0000", 2023-12-06T08:03:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054775470000000001 2023-12-06T08:03:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.934914+0000", 2023-12-06T08:03:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.116608303 2023-12-06T08:03:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.011339+0000", 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076425646 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.012363+0000", 2023-12-06T08:03:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010232990000000001 2023-12-06T08:03:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013117+0000", 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075423099999999996 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.65316820599998, 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18040658500000001, 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 33 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.624 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835156+0000", 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835169+0000", 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2612000000000001e-05 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835334+0000", 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165091 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835478+0000", 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014421400000000001 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.835622+0000", 2023-12-06T08:03:31.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014354400000000001 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.837754+0000", 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021321560000000001 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.839603+0000", 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001849072 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.842535+0000", 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029317900000000001 2023-12-06T08:03:31.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940148+0000", 2023-12-06T08:03:31.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097613247 2023-12-06T08:03:31.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.013914+0000", 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073766232000000001 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.014850+0000", 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093544400000000001 2023-12-06T08:03:31.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.015563+0000", 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071318300000000005 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.65167743500001, 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.181331467, 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 34 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836647+0000", 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836657+0000", 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0069e-05 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.836790+0000", 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013249600000000001 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850210+0000", 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013420437 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.850383+0000", 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017295200000000001 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.852995+0000", 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026117409999999999 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.854655+0000", 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001659879 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.857350+0000", 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026953559999999999 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940259+0000", 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082908765999999995 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.016306+0000", 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.076047371000000002 2023-12-06T08:03:31.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017281+0000", 2023-12-06T08:03:31.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097424299999999996 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.017979+0000", 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069815700000000003 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.61932777300001, 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160665529, 2023-12-06T08:03:31.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 35 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.868997+0000", 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.869010+0000", 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2952999999999999e-05 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.874640+0000", 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056303150000000003 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875163+0000", 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052332400000000003 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875326+0000", 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016237900000000001 2023-12-06T08:03:31.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.877924+0000", 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025979990000000001 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.879588+0000", 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016637830000000001 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.882833+0000", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003245235 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940376+0000", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.057543456999999999 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.027928+0000", 2023-12-06T08:03:31.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087551540999999997 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.028933+0000", 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001005358 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.029662+0000", 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072918500000000003 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.61269699799999, 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15722475499999999, 2023-12-06T08:03:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 36 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.654 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875628+0000", 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875638+0000", 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0839000000000001e-05 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875770+0000", 2023-12-06T08:03:31.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013114800000000001 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.875912+0000", 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014274099999999999 2023-12-06T08:03:31.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876049+0000", 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013707199999999999 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.889518+0000", 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01346846 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.891194+0000", 2023-12-06T08:03:31.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016756869999999999 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.893975+0000", 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027815890000000001 2023-12-06T08:03:31.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940475+0000", 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046499396999999998 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.030393+0000", 2023-12-06T08:03:31.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089918188999999996 2023-12-06T08:03:31.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.031701+0000", 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307927 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.032852+0000", 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001151706 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 369.61204631700002, 2023-12-06T08:03:31.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.160664789, 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.667 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 37 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:31.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876278+0000", 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876287+0000", 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5469999999999997e-06 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876439+0000", 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152399 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876573+0000", 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133621 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.876721+0000", 2023-12-06T08:03:31.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000147973 2023-12-06T08:03:31.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.899110+0000", 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022388755999999999 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.901993+0000", 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028832570000000002 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.906727+0000", 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047336440000000004 2023-12-06T08:03:31.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:21.940573+0000", 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033846635 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.033985+0000", 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093411868999999995 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.035728+0000", 2023-12-06T08:03:31.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017430900000000001 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.036943+0000", 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012149979999999999 2023-12-06T08:03:31.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 368.60857362600001, 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15485331599999999, 2023-12-06T08:03:31.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879751+0000", 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.879767+0000", 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5739999999999998e-05 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880036+0000", 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026897299999999999 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880277+0000", 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024124899999999999 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880508+0000", 2023-12-06T08:03:31.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231325 2023-12-06T08:03:31.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.882685+0000", 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021768349999999998 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.884269+0000", 2023-12-06T08:03:31.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015836540000000001 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.886923+0000", 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002654156 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981252+0000", 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094329017000000001 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.032910+0000", 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051657548999999997 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.033905+0000", 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000995695 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.034604+0000", 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069912299999999995 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 368.607434904, 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16188033499999999, 2023-12-06T08:03:31.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.691 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:31.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880890+0000", 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.880904+0000", 2023-12-06T08:03:31.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4218e-05 2023-12-06T08:03:31.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881100+0000", 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196236 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881301+0000", 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200471 2023-12-06T08:03:31.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881512+0000", 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021094899999999999 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.891148+0000", 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096364269999999995 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.892744+0000", 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001595539 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.895329+0000", 2023-12-06T08:03:31.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025855539999999999 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981366+0000", 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086037346000000001 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.041075+0000", 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.059708850000000001 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042066+0000", 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099043700000000004 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.042770+0000", 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070430799999999995 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 368.60648720699999, 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16335291499999999, 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881837+0000", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.881851+0000", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485e-05 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.908915+0000", 2023-12-06T08:03:31.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027063637000000001 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909179+0000", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026481200000000001 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.909417+0000", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000237338 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.911841+0000", 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024246049999999998 2023-12-06T08:03:31.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.914138+0000", 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022963649999999999 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.917389+0000", 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003251393 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981483+0000", 2023-12-06T08:03:31.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064094259000000001 2023-12-06T08:03:31.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.043508+0000", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.062024559 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.044469+0000", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096075000000000002 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045190+0000", 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000721712 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 368.57812321300003, 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13736610799999999, 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 85 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910201+0000", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910214+0000", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2959000000000001e-05 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910352+0000", 2023-12-06T08:03:31.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013727100000000001 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910620+0000", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026804299999999998 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.910763+0000", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000142948 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.925206+0000", 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014443064 2023-12-06T08:03:31.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.926797+0000", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001591035 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.929428+0000", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026316270000000001 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.981581+0000", 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052152970999999999 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.045907+0000", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.064325915999999997 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.046842+0000", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093426499999999999 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.047568+0000", 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072600900000000001 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.712 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 368.54991369599998, 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.117097798, 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 87 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938411+0000", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.938420+0000", 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6219999999999998e-06 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953237+0000", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014817877 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953382+0000", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144995 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.953526+0000", 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014306 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.954803+0000", 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012773579999999999 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.956456+0000", 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016530399999999999 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.959067+0000", 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026106200000000001 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:22.986981+0000", 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027914208999999999 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.053875+0000", 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.066894448999999995 2023-12-06T08:03:31.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.054836+0000", 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096058799999999996 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:23.055509+0000", 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067298 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 362.93971576199999, 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148398592, 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 105 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548609+0000", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548618+0000", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5030000000000003e-06 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548746+0000", 2023-12-06T08:03:31.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128069 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.548888+0000", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014107099999999999 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549033+0000", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014551899999999999 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.574632+0000", 2023-12-06T08:03:31.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025598775000000001 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.576684+0000", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020523859999999998 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.579661+0000", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002976575 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667438+0000", 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.08777712 2023-12-06T08:03:31.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.692585+0000", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025146980999999999 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.695950+0000", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033647429999999999 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.697007+0000", 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010578499999999999 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.723 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 362.93907092199998, 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15142181599999999, 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 106 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549254+0000", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.549263+0000", 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1039999999999996e-06 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.568740+0000", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019476884 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569023+0000", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028354399999999999 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569289+0000", 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026563900000000002 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.584038+0000", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014749115 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.585592+0000", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001554054 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.588075+0000", 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024826539999999999 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667616+0000", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079541412000000006 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.698400+0000", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030783748 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.699440+0000", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010404780000000001 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.700676+0000", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012351840000000001 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 362.918604168, 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13916916600000001, 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 107 2023-12-06T08:03:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569720+0000", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569738+0000", 2023-12-06T08:03:31.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7912999999999999e-05 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.569955+0000", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021703 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570096+0000", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014037199999999999 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570240+0000", 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014414699999999999 2023-12-06T08:03:31.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.608522+0000", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038282203000000001 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.610206+0000", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001683422 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.612854+0000", 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026482739999999999 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.667811+0000", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.054957513999999999 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.705322+0000", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037510786999999997 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708128+0000", 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028063340000000002 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.708890+0000", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076117000000000003 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 362.91785525199998, 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095575799999999, 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.734 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 108 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570469+0000", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570479+0000", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2050000000000008e-06 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570603+0000", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012447399999999999 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570737+0000", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013376499999999999 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.570883+0000", 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145821 2023-12-06T08:03:31.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.617167+0000", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046284277999999998 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.618737+0000", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001570519 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.621289+0000", 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025520349999999998 2023-12-06T08:03:31.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.668008+0000", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046718535999999998 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.709724+0000", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.041716151 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.710715+0000", 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099034899999999996 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.711425+0000", 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071062499999999999 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4170.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 362.917213183, 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14278113200000001, 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:31.739 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4170", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2639546867", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571111+0000", 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.571121+0000", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0599999999999997e-06 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.603961+0000", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032840925 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604294+0000", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033228499999999998 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:31.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.604550+0000", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025607199999999998 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.625240+0000", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020690281000000001 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.626725+0000", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014853970000000001 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2023-12-06T08:03:31.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.629252+0000", 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025266720000000002 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.669026+0000", 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039774219999999999 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2023-12-06T08:03:31.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.712216+0000", 2023-12-06T08:03:31.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043189943000000001 2023-12-06T08:03:31.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713187+0000", 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097037200000000005 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:28.713893+0000", 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070590500000000005 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:31.745 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:31.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-12-06T08:03:31.747 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:31.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:31.748 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:13.321 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:31.748 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:13.323 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:31.748 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:13.372 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:31.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:31.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:31.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:31.750 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:31.747+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:31.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:03:31.847 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:31.847+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:31.944 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:31.944 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:31.944 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:31.944 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:31.948 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:31.948+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:31.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-12-06T08:03:32.049 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:32.049+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:32.124 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:32.125 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:32.125 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:32.125 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:32.125 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:32.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-12-06T08:03:32.176 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:32.175+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:32.250 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:32.250+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:32.304 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2023-12-06T08:03:32.304 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:32.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-12-06T08:03:32.356 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:32.356+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:32.452 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:32.451+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:32.495 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:32.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:32.496 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2023-12-06T08:03:32.496 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:32.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-12-06T08:03:32.553 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:32.552+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:32.653 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:32.653+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:32.674 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2023-12-06T08:03:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2023-12-06T08:03:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2023-12-06T08:03:32.674 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:32.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-12-06T08:03:32.768 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:32.768+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:32.855 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:03:32.855+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.44027622200002, 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079627879999999998, 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:32.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415467+0000", 2023-12-06T08:03:32.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.415662+0000", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019536900000000001 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.420058+0000", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043959719999999997 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.424101+0000", 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040425799999999996 2023-12-06T08:03:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.432606+0000", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085051339999999993 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.442251+0000", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009644662 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T07:57:39.495095+0000", 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052844163 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.864 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:7 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.27677745700001, 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045302140000000003, 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.865 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578966+0000", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.578979+0000", 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3694e-05 2023-12-06T08:03:32.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579134+0000", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015455199999999999 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579315+0000", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000181103 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.579481+0000", 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016612800000000001 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.582053+0000", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025716390000000001 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:00:39.583496+0000", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014430980000000001 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.4176.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:32.868 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 113.26971644299999, 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042358229999999997, 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.4176", 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586027+0000", 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586042+0000", 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5129e-05 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586227+0000", 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018534500000000001 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586439+0000", 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021191 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2023-12-06T08:03:32.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.586627+0000", 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018811699999999999 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.588755+0000", 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021280000000000001 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2023-12-06T08:01:39.590263+0000", 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001507322 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2023-12-06T08:03:32.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2023-12-06T08:03:32.873 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:03:32.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-12-06T08:03:32.956 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:03:32.956+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:03:33.005 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.005 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:33.005 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:33.005 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-12-06T08:03:33.057 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:33.056+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:33.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-06T08:03:33.158 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:33.158+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:33.210 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.210 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:33.210 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:33.210 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:33.210 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-12-06T08:03:33.258 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:33.258+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:33.347 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:33.347 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-12-06T08:03:33.359 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:33.359+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:33.459 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:03:33.459+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:03:33.489 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:33.489 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-12-06T08:03:33.560 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:03:33.560+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:03:33.619 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:03:33.655 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:33.656 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-12-06T08:03:33.675 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:03:33.675+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:03:33.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-06T08:03:33.795 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:03:33.795+0000 b47e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:03:33.807 INFO:teuthology.orchestra.run.smithi084.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","modified":"2023-12-06T07:55:36.076779Z","created":"2023-12-06T07:55:36.076779Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-06T08:03:33.817 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2023-12-06T08:03:33.818 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-12-06T08:03:33.833 INFO:tasks.thrashosds:joining thrashosds 2023-12-06T08:03:33.845 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2023-12-06T08:03:33.845 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:33.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-12-06T08:03:33.994 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:33.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:33.994 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:03:33.994 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:34.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-12-06T08:03:34.027 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2023-12-06T08:03:34.028 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2023-12-06T08:03:34.109 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2023-12-06T08:03:34.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2023-12-06T08:03:34.132 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:34.132 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:03:34.132 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2023-12-06T08:03:34.132 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2023-12-06T08:03:34.132 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:34.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-12-06T08:03:34.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2023-12-06T08:03:34.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2023-12-06T08:03:34.274 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 353.69489968200003, 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091966784999999995, 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:34.275 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:34.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.574930+0000", 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.575131+0000", 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000201545 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.579562+0000", 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306789999999999 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.583696+0000", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041341420000000004 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.592739+0000", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090424860000000006 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.612444+0000", 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019705555 2023-12-06T08:03:34.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T07:57:40.666896+0000", 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054452378000000003 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:8 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 173.57772233599999, 2023-12-06T08:03:34.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004823712, 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692107+0000", 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692123+0000", 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6124e-05 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692299+0000", 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017538499999999999 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692493+0000", 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019417600000000001 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.692663+0000", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169949 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.695515+0000", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028526440000000001 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:00:40.696931+0000", 2023-12-06T08:03:34.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014154339999999999 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4176.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 113.57091812, 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031235120000000002, 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2023-12-06T08:03:34.284 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4176", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/2060349495", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698911+0000", 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2023-12-06T08:03:34.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.698925+0000", 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3524e-05 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699089+0000", 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164458 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2023-12-06T08:03:34.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699278+0000", 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188321 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.699444+0000", 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016656400000000001 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.700597+0000", 2023-12-06T08:03:34.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011528529999999999 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2023-12-06T08:01:40.702035+0000", 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437792 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-12-06T08:03:34.289 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-12-06T08:03:34.289 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:03:34.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2023-12-06T08:03:34.290 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2023-12-06T08:03:34.369 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2023-12-06T08:03:34.369 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2023-12-06T08:03:34.371 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2023-12-06T08:03:34.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2023-12-06T08:03:34.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2023-12-06T08:03:34.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2023-12-06T08:03:34.549 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2023-12-06T08:03:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-06T08:03:34.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-06T08:03:35.613 INFO:tasks.thrashosds.thrasher:reviving osd 2023-12-06T08:03:35.613 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2023-12-06T08:03:35.613 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-06T08:03:35.613 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-06T08:03:35.615 INFO:tasks.ceph.osd.5:Started 2023-12-06T08:03:35.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:35.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-12-06T08:03:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:03:35.714 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T08:03:35.714 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:03:37.258 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2023-12-06T08:03:37.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-12-06T08:03:38.390 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2023-12-06T08:03:39.876 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:36.309 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.877 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:36.310 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.897 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:36.330 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.920 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:33.874 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.928 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:33.880 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.934 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:21.598 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.938 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:21.601 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.946 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:33.900 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:39.951 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:21.615 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:40.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:40.811 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T08:03:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:03:40.814 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:03:43.151 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:07.472 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:43.207 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:07.528 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:43.772 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:03:43.766+0000 5dc2340 -1 Falling back to public interface 2023-12-06T08:03:44.174 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:08.493 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:44.529 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:08.849 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:44.840 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.161 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:44.883 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:41.316 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.890 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:41.320 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.900 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:02:41.333 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.931 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:38.885 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.934 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:38.888 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.940 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:26.604 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.942 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:26.606 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.945 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:38.899 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:44.954 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:26.618 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:03:45.126 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.447 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:45.418 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:09.740 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:45.706 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.027 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:45.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:0 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:1 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:2 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2023-12-06T08:03:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:config diff 2023-12-06T08:03:46.000 INFO:teuthology.orchestra.run.smithi107.stderr:config diff get 2023-12-06T08:03:46.000 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2023-12-06T08:03:46.000 INFO:teuthology.orchestra.run.smithi107.stderr:config help [] 2023-12-06T08:03:46.000 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2023-12-06T08:03:46.000 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2023-12-06T08:03:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:03:46.001 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:03:46.014 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.335 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:46.056 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.377 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:46.351 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:10.672 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:46.861 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:11.182 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:46.913 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:11.234 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:51.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:0 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:1 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:2 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation histogram block [] [] 2023-12-06T08:03:51.768 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator score block 2023-12-06T08:03:51.769 INFO:teuthology.orchestra.run.smithi107.stderr:config diff 2023-12-06T08:03:51.769 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2023-12-06T08:03:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:03:51.770 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:03:51.868 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:16.189 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:56.309 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:20.630 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:56.312 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:20.634 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:03:56.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:0 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:1 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2023-12-06T08:03:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore bluefs device info [] 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:config diff 2023-12-06T08:03:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2023-12-06T08:03:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:03:58.670 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:03:59.956 INFO:tasks.ceph.osd.5.smithi107.stderr:==00:00:00:24.277 113651== Warning: unimplemented fcntl command: 1038 2023-12-06T08:04:03.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:0 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:1 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:2 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2023-12-06T08:04:04.284 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2023-12-06T08:04:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2023-12-06T08:04:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-06T08:04:04.287 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2023-12-06T08:04:04.931 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:01.363 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.932 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:01.365 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.947 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:01.380 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.977 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:58.931 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.980 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:46.643 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.980 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:58.934 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.983 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:46.647 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.993 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:01:58.947 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:04.998 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:46.662 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:07.217 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:04:07.215+0000 5dc2340 -1 osd.5 93 log_to_monitors true 2023-12-06T08:04:09.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-12-06T08:04:09.827 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-12-06T08:04:09.827 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2023-12-06T08:04:09.827 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2023-12-06T08:04:09.827 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-12-06T08:04:09.836 INFO:tasks.thrashosds.thrasher:inning osd 2023-12-06T08:04:09.836 INFO:tasks.thrashosds.thrasher:Adding osd 7 2023-12-06T08:04:09.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2023-12-06T08:04:09.839 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:04:09.828+0000 13a5a640 -1 osd.5 107 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T08:04:10.646 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:04:10.643+0000 ff5f640 -1 mon.a@0(leader).osd e107 definitely_dead 0 2023-12-06T08:04:11.096 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.7. 2023-12-06T08:04:11.112 INFO:tasks.thrashosds.thrasher:Added osd 7 2023-12-06T08:04:11.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2023-12-06T08:04:11.412 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2023-12-06T08:04:11.412 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T08:04:11.551 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2023-12-06T08:04:11.552 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T08:04:11.665 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2023-12-06T08:04:11.665 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-06T08:04:11.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-06T08:04:13.354 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2023-12-06T08:04:13.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2023-12-06T08:04:14.482 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2023-12-06T08:04:14.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-06T08:04:15.689 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2023-12-06T08:04:15.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2023-12-06T08:04:16.662 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 1 (8655362) 2023-12-06T08:04:16.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-06T08:04:17.743 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2023-12-06T08:04:17.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-12-06T08:04:18.919 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-12-06T08:04:18.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-06T08:04:20.000 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2023-12-06T08:04:20.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2023-12-06T08:04:21.163 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.3 primary-affinity to 1 (8655362) 2023-12-06T08:04:21.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-12-06T08:04:22.253 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2023-12-06T08:04:22.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2023-12-06T08:04:23.398 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 1 (8655362) 2023-12-06T08:04:23.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-12-06T08:04:24.445 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2023-12-06T08:04:24.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2023-12-06T08:04:25.004 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:21.436 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.009 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:21.440 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.035 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:21.467 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.051 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:19.005 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.053 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:19.007 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.056 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:06.720 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.062 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:06.722 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.076 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:19.030 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:25.090 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:06.754 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:26.188 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.6 primary-affinity to 1 (8655362) 2023-12-06T08:04:26.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-12-06T08:04:27.645 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2023-12-06T08:04:27.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2023-12-06T08:04:28.714 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 1 (8655362) 2023-12-06T08:04:28.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-12-06T08:04:29.912 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2023-12-06T08:04:29.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2023-12-06T08:04:31.189 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2023-12-06T08:04:31.203 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-12-06T08:04:31.203 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-12-06T08:04:31.203 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-12-06T08:04:31.203 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-12-06T08:04:31.203 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-06T08:04:31.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T08:04:31.881 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:31.881 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":126,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T08:04:31.053345+0000","last_up_change":"2023-12-06T08:04:10.917942+0000","last_in_change":"2023-12-06T08:04:10.917942+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":93,"up_from":108,"up_thru":108,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3236700828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3236700828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3236700828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":3236700828}]},"public_addr":"172.21.15.107:6809/3236700828","cluster_addr":"172.21.15.107:6810/3236700828","heartbeat_back_addr":"172.21.15.107:6812/3236700828","heartbeat_front_addr":"172.21.15.107:6811/3236700828","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":93},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[4,3]},{"pgid":"2.3","osds":[0,3]},{"pgid":"2.5","osds":[1,6]},{"pgid":"2.6","osds":[7,2]},{"pgid":"2.7","osds":[1,7]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T08:04:31.895 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-06T08:04:31.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T08:04:32.489 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:32.489 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":126,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T08:04:31.053345+0000","last_up_change":"2023-12-06T08:04:10.917942+0000","last_in_change":"2023-12-06T08:04:10.917942+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":93,"up_from":108,"up_thru":108,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3236700828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3236700828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3236700828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":3236700828}]},"public_addr":"172.21.15.107:6809/3236700828","cluster_addr":"172.21.15.107:6810/3236700828","heartbeat_back_addr":"172.21.15.107:6812/3236700828","heartbeat_front_addr":"172.21.15.107:6811/3236700828","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":93},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[4,3]},{"pgid":"2.3","osds":[0,3]},{"pgid":"2.5","osds":[1,6]},{"pgid":"2.6","osds":[7,2]},{"pgid":"2.7","osds":[1,7]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T08:04:32.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-06T08:04:32.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-06T08:04:32.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-06T08:04:32.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-06T08:04:32.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-06T08:04:32.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-06T08:04:32.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-06T08:04:32.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-06T08:04:32.852 INFO:teuthology.orchestra.run.smithi084.stdout:68719476829 2023-12-06T08:04:32.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T08:04:32.853 INFO:teuthology.orchestra.run.smithi084.stdout:68719476827 2023-12-06T08:04:32.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T08:04:32.855 INFO:teuthology.orchestra.run.smithi084.stdout:60129542235 2023-12-06T08:04:32.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T08:04:32.886 INFO:teuthology.orchestra.run.smithi084.stdout:64424509532 2023-12-06T08:04:32.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T08:04:32.901 INFO:teuthology.orchestra.run.smithi084.stdout:68719476827 2023-12-06T08:04:32.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T08:04:32.922 INFO:teuthology.orchestra.run.smithi084.stdout:463856467974 2023-12-06T08:04:32.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T08:04:32.934 INFO:teuthology.orchestra.run.smithi084.stdout:55834574938 2023-12-06T08:04:32.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T08:04:32.949 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:32.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T08:04:33.597 INFO:teuthology.orchestra.run.smithi084.stdout:60129542233 2023-12-06T08:04:33.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542235 got 60129542233 for osd.6 2023-12-06T08:04:33.814 INFO:teuthology.orchestra.run.smithi084.stdout:463856467973 2023-12-06T08:04:33.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467974 got 463856467973 for osd.5 2023-12-06T08:04:34.007 INFO:teuthology.orchestra.run.smithi084.stdout:55834574937 2023-12-06T08:04:34.010 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:34.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476826 for osd.1 2023-12-06T08:04:34.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574938 got 55834574937 for osd.7 2023-12-06T08:04:34.079 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:34.082 INFO:teuthology.orchestra.run.smithi084.stdout:64424509531 2023-12-06T08:04:34.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476826 for osd.2 2023-12-06T08:04:34.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509532 got 64424509531 for osd.4 2023-12-06T08:04:34.220 INFO:teuthology.orchestra.run.smithi084.stdout:68719476825 2023-12-06T08:04:34.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476826 got 68719476825 for osd.3 2023-12-06T08:04:34.257 INFO:teuthology.orchestra.run.smithi084.stdout:68719476828 2023-12-06T08:04:34.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476829 got 68719476828 for osd.0 2023-12-06T08:04:34.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T08:04:34.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T08:04:35.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T08:04:35.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T08:04:35.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T08:04:35.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T08:04:35.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-06T08:04:35.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-06T08:04:35.431 INFO:teuthology.orchestra.run.smithi084.stdout:60129542233 2023-12-06T08:04:35.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542235 got 60129542233 for osd.6 2023-12-06T08:04:35.692 INFO:teuthology.orchestra.run.smithi084.stdout:463856467973 2023-12-06T08:04:35.707 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467974 got 463856467973 for osd.5 2023-12-06T08:04:35.732 INFO:teuthology.orchestra.run.smithi084.stdout:55834574937 2023-12-06T08:04:35.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574938 got 55834574937 for osd.7 2023-12-06T08:04:35.899 INFO:teuthology.orchestra.run.smithi084.stdout:64424509531 2023-12-06T08:04:35.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509532 got 64424509531 for osd.4 2023-12-06T08:04:35.996 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:36.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476826 for osd.1 2023-12-06T08:04:36.061 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:36.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476826 for osd.2 2023-12-06T08:04:36.143 INFO:teuthology.orchestra.run.smithi084.stdout:68719476826 2023-12-06T08:04:36.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476826 got 68719476826 for osd.3 2023-12-06T08:04:36.157 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:36.196 INFO:teuthology.orchestra.run.smithi084.stdout:68719476829 2023-12-06T08:04:36.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476829 got 68719476829 for osd.0 2023-12-06T08:04:36.211 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:36.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-06T08:04:36.708 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-06T08:04:36.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-06T08:04:36.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-06T08:04:37.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-06T08:04:37.046 INFO:teuthology.orchestra.run.smithi084.stdout:60129542235 2023-12-06T08:04:37.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542235 got 60129542235 for osd.6 2023-12-06T08:04:37.064 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-06T08:04:37.461 INFO:teuthology.orchestra.run.smithi084.stdout:463856467974 2023-12-06T08:04:37.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467974 got 463856467974 for osd.5 2023-12-06T08:04:37.480 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.540 INFO:teuthology.orchestra.run.smithi084.stdout:55834574938 2023-12-06T08:04:37.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574938 got 55834574938 for osd.7 2023-12-06T08:04:37.559 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.652 INFO:teuthology.orchestra.run.smithi084.stdout:64424509533 2023-12-06T08:04:37.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509532 got 64424509533 for osd.4 2023-12-06T08:04:37.669 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.702 INFO:teuthology.orchestra.run.smithi084.stdout:68719476827 2023-12-06T08:04:37.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476827 for osd.1 2023-12-06T08:04:37.720 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.737 INFO:teuthology.orchestra.run.smithi084.stdout:68719476827 2023-12-06T08:04:37.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476827 got 68719476827 for osd.2 2023-12-06T08:04:37.753 DEBUG:teuthology.parallel:result is None 2023-12-06T08:04:37.753 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-06T08:04:37.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:38.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:38.348 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:38.361 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-12-06T08:04:37.876386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229056,"kb_used_data":7664,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745664,"statfs":{"total":773094113280,"available":772859559936,"internally_reserved":0,"allocated":7847936,"data_stored":3367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.885270"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.113198+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:31.113198+0000","last_peered":"2023-12-06T08:04:31.113198+0000","last_clean":"2023-12-06T08:04:31.113198+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:31.113198+0000","last_undegraded":"2023-12-06T08:04:31.113198+0000","last_fullsized":"2023-12-06T08:04:31.113198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:55:49.691370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.054362+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:30.054362+0000","last_peered":"2023-12-06T08:04:30.054362+0000","last_clean":"2023-12-06T08:04:30.054362+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:30.054362+0000","last_undegraded":"2023-12-06T08:04:30.054362+0000","last_fullsized":"2023-12-06T08:04:30.054362+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.110729+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:31.110729+0000","last_peered":"2023-12-06T08:04:31.110729+0000","last_clean":"2023-12-06T08:04:31.110729+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:31.110729+0000","last_undegraded":"2023-12-06T08:04:31.110729+0000","last_fullsized":"2023-12-06T08:04:31.110729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":83,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.004771+0000","last_change":"2023-12-06T08:04:12.508710+0000","last_active":"2023-12-06T08:04:30.004771+0000","last_peered":"2023-12-06T08:04:30.004771+0000","last_clean":"2023-12-06T08:04:30.004771+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:30.004771+0000","last_undegraded":"2023-12-06T08:04:30.004771+0000","last_fullsized":"2023-12-06T08:04:30.004771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:22:19.396954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.100874+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.100874+0000","last_peered":"2023-12-06T08:04:31.100874+0000","last_clean":"2023-12-06T08:04:31.100874+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.100874+0000","last_undegraded":"2023-12-06T08:04:31.100874+0000","last_fullsized":"2023-12-06T08:04:31.100874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.050973+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:30.050973+0000","last_peered":"2023-12-06T08:04:30.050973+0000","last_clean":"2023-12-06T08:04:30.050973+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:30.050973+0000","last_undegraded":"2023-12-06T08:04:30.050973+0000","last_fullsized":"2023-12-06T08:04:30.050973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.114484+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:31.114484+0000","last_peered":"2023-12-06T08:04:31.114484+0000","last_clean":"2023-12-06T08:04:31.114484+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:31.114484+0000","last_undegraded":"2023-12-06T08:04:31.114484+0000","last_fullsized":"2023-12-06T08:04:31.114484+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.005658+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:30.005658+0000","last_peered":"2023-12-06T08:04:30.005658+0000","last_clean":"2023-12-06T08:04:30.005658+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:30.005658+0000","last_undegraded":"2023-12-06T08:04:30.005658+0000","last_fullsized":"2023-12-06T08:04:30.005658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:01:11.918493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":234,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.108949+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:31.108949+0000","last_peered":"2023-12-06T08:04:31.108949+0000","last_clean":"2023-12-06T08:04:31.108949+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:31.108949+0000","last_undegraded":"2023-12-06T08:04:31.108949+0000","last_fullsized":"2023-12-06T08:04:31.108949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:55:38.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":17.556000000000001},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":11.894}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.2609999999999992},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":7.8019999999999996}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":12.472},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":7.3029999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":11.275},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.852}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":16.565999999999999},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.449}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.058},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":10.981}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":18.334},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":14.66}]}]},{"osd":2,"up_from":16,"seq":68719476828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":9.7089999999999996,"15min":10.234},"min":{"1min":4.9109999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":40.771999999999998,"5min":40.771999999999998,"15min":40.771999999999998},"last":12.353999999999999},{"interface":"front","average":{"1min":9.8580000000000005,"5min":9.8480000000000008,"15min":10.356},"min":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":45.781999999999996,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.665}]},{"osd":1,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.401,"5min":12.720000000000001,"15min":19.236000000000001},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":21.439,"5min":27.763000000000002,"15min":165.68600000000001},"last":13.706},{"interface":"front","average":{"1min":11.988,"5min":12.721,"15min":19.760999999999999},"min":{"1min":7.8300000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":17.602,"5min":27.974,"15min":153.68199999999999},"last":9.2010000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.388,"5min":10.52,"15min":11.021000000000001},"min":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":23.768999999999998,"5min":26.887,"15min":35.935000000000002},"last":15.625},{"interface":"front","average":{"1min":14.199999999999999,"5min":15.276,"15min":15.566000000000001},"min":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":25.352,"5min":30.433,"15min":30.433},"last":14.285}]},{"osd":4,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":13.138,"15min":15.73},"min":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":34.514000000000003,"5min":48.207999999999998,"15min":50.137999999999998},"last":16.573},{"interface":"front","average":{"1min":17.808,"5min":17.616,"15min":16.946000000000002},"min":{"1min":10.286,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":51.164999999999999,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.154}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":18.66},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":17.364000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":13.91,"5min":15.550000000000001,"15min":18.908000000000001},"min":{"1min":7.71,"5min":7.71,"15min":7.71},"max":{"1min":24.242000000000001,"5min":48.231000000000002,"15min":104.834},"last":20.071000000000002},{"interface":"front","average":{"1min":17.329000000000001,"5min":19.853999999999999,"15min":20.48},"min":{"1min":11.135,"5min":11.135,"15min":11.135},"max":{"1min":25.175000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.388000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.974,"5min":16.959,"15min":18.948},"min":{"1min":8.1270000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":34.999000000000002,"5min":34.999000000000002,"15min":42.915999999999997},"last":10.621},{"interface":"front","average":{"1min":17.074000000000002,"5min":17.196999999999999,"15min":18.687000000000001},"min":{"1min":9.2289999999999992,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":33.512999999999998,"5min":43.697000000000003,"15min":43.697000000000003},"last":21.213000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":12.513,"5min":10.885,"15min":12.747999999999999},"min":{"1min":5.476,"5min":4.835,"15min":4.835},"max":{"1min":28.488,"5min":36.534999999999997,"15min":80.640000000000001},"last":23.114999999999998},{"interface":"front","average":{"1min":15.066000000000001,"5min":12.802,"15min":12.042},"min":{"1min":8.7309999999999999,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":26.504000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":25.140999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":16.111000000000001,"5min":14.246,"15min":18.893000000000001},"min":{"1min":10.234999999999999,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.82,"5min":32.081000000000003,"15min":100.703},"last":24.52},{"interface":"front","average":{"1min":9.2789999999999999,"5min":10.380000000000001,"15min":16.532},"min":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"max":{"1min":29.946999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.5919999999999996}]},{"osd":3,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.788,"5min":16.448,"15min":17.936},"min":{"1min":11.882999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":44.055,"5min":44.055,"15min":44.055},"last":26.606999999999999},{"interface":"front","average":{"1min":10.44,"5min":10.215,"15min":12.718999999999999},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":42.786000000000001,"5min":42.786000000000001,"15min":46.613},"last":6.0389999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.440000000000001,"5min":15.339,"15min":17.645},"min":{"1min":12.68,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":81.838999999999999},"last":27.620999999999999},{"interface":"front","average":{"1min":19.515999999999998,"5min":18.312999999999999,"15min":20.640999999999998},"min":{"1min":15.148,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":33.311,"5min":33.311,"15min":66.515000000000001},"last":28.974}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":9.8900000000000006},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":29.850999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":13.477,"5min":14.614000000000001,"15min":17.489999999999998},"min":{"1min":8.6739999999999995,"5min":8.2789999999999999,"15min":8.2789999999999999},"max":{"1min":19.747,"5min":31.954999999999998,"15min":58.619},"last":17.129999999999999},{"interface":"front","average":{"1min":12.161,"5min":14.904,"15min":19.318999999999999},"min":{"1min":5.8769999999999998,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":25.501999999999999,"5min":33.671999999999997,"15min":83.632000000000005},"last":21.196000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.297999999999998,"15min":21.789000000000001},"min":{"1min":12.069000000000001,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":34.576999999999998,"5min":34.576999999999998,"15min":62.006999999999998},"last":32.790999999999997},{"interface":"front","average":{"1min":13.202999999999999,"5min":15.308,"15min":19.100999999999999},"min":{"1min":8.9480000000000004,"5min":8.0719999999999992,"15min":8.0719999999999992},"max":{"1min":22.373000000000001,"5min":32.512999999999998,"15min":60.698999999999998},"last":19.228000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":10.68,"5min":10.488,"15min":17.312000000000001},"min":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":25.873000000000001,"5min":64.242999999999995,"15min":252.88399999999999},"last":8.1430000000000007},{"interface":"front","average":{"1min":12.551,"5min":11.323,"15min":18.364999999999998},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":33.127000000000002,"5min":63.192999999999998,"15min":274.91500000000002},"last":13.922000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.704000000000001,"5min":12.589,"15min":16.036999999999999},"min":{"1min":6.9960000000000004,"5min":5.093,"15min":5.093},"max":{"1min":23.831,"5min":35.383000000000003,"15min":82.866},"last":9.2059999999999995},{"interface":"front","average":{"1min":13.122,"5min":13.996,"15min":15.718},"min":{"1min":7.5750000000000002,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.742999999999999,"5min":34.289999999999999,"15min":85.991},"last":17.562999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.128,"5min":11.25,"15min":14.057},"min":{"1min":7.6230000000000002,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":24.594000000000001,"5min":41.045000000000002,"15min":76.593000000000004},"last":14.737},{"interface":"front","average":{"1min":16.071999999999999,"5min":15.656000000000001,"15min":19.260999999999999},"min":{"1min":10.727,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":26.637,"5min":42.284999999999997,"15min":85.162000000000006},"last":15.704000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":14.734,"5min":15.743,"15min":16.966000000000001},"min":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":24.876000000000001,"5min":76.387,"15min":76.387},"last":18.146999999999998},{"interface":"front","average":{"1min":13.502000000000001,"5min":13.249000000000001,"15min":13.815},"min":{"1min":7.7140000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":28.337,"5min":29.375,"15min":70.085999999999999},"last":16.645}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":19.116},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":12.494}]},{"osd":6,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":16.324999999999999,"15min":17.731000000000002},"min":{"1min":9.6999999999999993,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.141999999999999,"5min":31.550000000000001,"15min":75.471999999999994},"last":19.756},{"interface":"front","average":{"1min":18.029,"5min":17.558,"15min":22.172000000000001},"min":{"1min":13.818,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":30.916,"5min":37.479999999999997,"15min":84.617000000000004},"last":20.420999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":19.414999999999999,"5min":19.709,"15min":21.390000000000001},"min":{"1min":9.6750000000000007,"5min":9.6750000000000007,"15min":7.6219999999999999},"max":{"1min":48.314,"5min":78.506,"15min":84.031000000000006},"last":22.494},{"interface":"front","average":{"1min":19.506,"5min":17.164000000000001,"15min":18.596},"min":{"1min":12.209,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":49.362000000000002,"5min":49.362000000000002,"15min":77.974000000000004},"last":21.628}]}]},{"osd":4,"up_from":15,"seq":64424509533,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":10.154,"5min":8.6099999999999994,"15min":9.3480000000000008},"min":{"1min":4.4640000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":23.513000000000002,"5min":48.454000000000001,"15min":48.454000000000001},"last":14.603999999999999},{"interface":"front","average":{"1min":8.609,"5min":9.8610000000000007,"15min":10.148},"min":{"1min":5.0220000000000002,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":21.257000000000001,"5min":44.947000000000003,"15min":44.947000000000003},"last":4.5119999999999996}]},{"osd":1,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":10.202,"15min":11.420999999999999},"min":{"1min":5.9569999999999999,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":15.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":10.468},{"interface":"front","average":{"1min":12.343,"5min":12.006,"15min":13.369999999999999},"min":{"1min":7.149,"5min":6.5599999999999996,"15min":5.6269999999999998},"max":{"1min":25.908000000000001,"5min":26.003,"15min":31.027000000000001},"last":8.6440000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":13.532,"15min":12.522},"min":{"1min":7.9640000000000004,"5min":7.1379999999999999,"15min":7.1379999999999999},"max":{"1min":16.992000000000001,"5min":29.806999999999999,"15min":29.806999999999999},"last":19.222999999999999},{"interface":"front","average":{"1min":12.406000000000001,"5min":14.375,"15min":14.542},"min":{"1min":8.3629999999999995,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.27,"5min":32.915999999999997,"15min":32.915999999999997},"last":16.728999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":12.792999999999999,"5min":12.470000000000001,"15min":11.813000000000001},"min":{"1min":6.1180000000000003,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":24.111000000000001,"5min":28.991,"15min":28.991},"last":15.266999999999999},{"interface":"front","average":{"1min":15.817,"5min":15.462999999999999,"15min":14.531000000000001},"min":{"1min":9.1270000000000007,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":22.725000000000001,"5min":35.540999999999997,"15min":35.540999999999997},"last":11.093999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":13.359},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":7.891}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":17.841000000000001},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":19.827999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":13.946999999999999},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":18.460000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":12.375},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":7.1559999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":9.0570000000000004},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":6.2510000000000003}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":17.114999999999998},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":16.207999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.8539999999999992},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":15.446999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":37.387},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":38.234000000000002}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":14.472}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":11.664},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":13.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467975,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":1252,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.866}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.484000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.382999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.41}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.013999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.27}]}]},{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":18.798999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":15.212}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.4119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":12.298}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":13.695},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":18.061}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":15.82},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":20.184000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":19.553000000000001},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.619}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.649000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":14.57}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:38.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:38.947 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:38.947 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:38.960 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-12-06T08:04:37.876386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229056,"kb_used_data":7664,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745664,"statfs":{"total":773094113280,"available":772859559936,"internally_reserved":0,"allocated":7847936,"data_stored":3367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.885270"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.113198+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:31.113198+0000","last_peered":"2023-12-06T08:04:31.113198+0000","last_clean":"2023-12-06T08:04:31.113198+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:31.113198+0000","last_undegraded":"2023-12-06T08:04:31.113198+0000","last_fullsized":"2023-12-06T08:04:31.113198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:55:49.691370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.054362+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:30.054362+0000","last_peered":"2023-12-06T08:04:30.054362+0000","last_clean":"2023-12-06T08:04:30.054362+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:30.054362+0000","last_undegraded":"2023-12-06T08:04:30.054362+0000","last_fullsized":"2023-12-06T08:04:30.054362+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.110729+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:31.110729+0000","last_peered":"2023-12-06T08:04:31.110729+0000","last_clean":"2023-12-06T08:04:31.110729+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:31.110729+0000","last_undegraded":"2023-12-06T08:04:31.110729+0000","last_fullsized":"2023-12-06T08:04:31.110729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":83,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.004771+0000","last_change":"2023-12-06T08:04:12.508710+0000","last_active":"2023-12-06T08:04:30.004771+0000","last_peered":"2023-12-06T08:04:30.004771+0000","last_clean":"2023-12-06T08:04:30.004771+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:30.004771+0000","last_undegraded":"2023-12-06T08:04:30.004771+0000","last_fullsized":"2023-12-06T08:04:30.004771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:22:19.396954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.100874+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.100874+0000","last_peered":"2023-12-06T08:04:31.100874+0000","last_clean":"2023-12-06T08:04:31.100874+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.100874+0000","last_undegraded":"2023-12-06T08:04:31.100874+0000","last_fullsized":"2023-12-06T08:04:31.100874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.050973+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:30.050973+0000","last_peered":"2023-12-06T08:04:30.050973+0000","last_clean":"2023-12-06T08:04:30.050973+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:30.050973+0000","last_undegraded":"2023-12-06T08:04:30.050973+0000","last_fullsized":"2023-12-06T08:04:30.050973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.114484+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:31.114484+0000","last_peered":"2023-12-06T08:04:31.114484+0000","last_clean":"2023-12-06T08:04:31.114484+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:31.114484+0000","last_undegraded":"2023-12-06T08:04:31.114484+0000","last_fullsized":"2023-12-06T08:04:31.114484+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.005658+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:30.005658+0000","last_peered":"2023-12-06T08:04:30.005658+0000","last_clean":"2023-12-06T08:04:30.005658+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:30.005658+0000","last_undegraded":"2023-12-06T08:04:30.005658+0000","last_fullsized":"2023-12-06T08:04:30.005658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:01:11.918493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":234,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.108949+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:31.108949+0000","last_peered":"2023-12-06T08:04:31.108949+0000","last_clean":"2023-12-06T08:04:31.108949+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:31.108949+0000","last_undegraded":"2023-12-06T08:04:31.108949+0000","last_fullsized":"2023-12-06T08:04:31.108949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:55:38.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":17.556000000000001},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":11.894}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.2609999999999992},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":7.8019999999999996}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":12.472},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":7.3029999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":11.275},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.852}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":16.565999999999999},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.449}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.058},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":10.981}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":18.334},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":14.66}]}]},{"osd":2,"up_from":16,"seq":68719476828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":9.7089999999999996,"15min":10.234},"min":{"1min":4.9109999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":40.771999999999998,"5min":40.771999999999998,"15min":40.771999999999998},"last":12.353999999999999},{"interface":"front","average":{"1min":9.8580000000000005,"5min":9.8480000000000008,"15min":10.356},"min":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":45.781999999999996,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.665}]},{"osd":1,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.401,"5min":12.720000000000001,"15min":19.236000000000001},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":21.439,"5min":27.763000000000002,"15min":165.68600000000001},"last":13.706},{"interface":"front","average":{"1min":11.988,"5min":12.721,"15min":19.760999999999999},"min":{"1min":7.8300000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":17.602,"5min":27.974,"15min":153.68199999999999},"last":9.2010000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.388,"5min":10.52,"15min":11.021000000000001},"min":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":23.768999999999998,"5min":26.887,"15min":35.935000000000002},"last":15.625},{"interface":"front","average":{"1min":14.199999999999999,"5min":15.276,"15min":15.566000000000001},"min":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":25.352,"5min":30.433,"15min":30.433},"last":14.285}]},{"osd":4,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":13.138,"15min":15.73},"min":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":34.514000000000003,"5min":48.207999999999998,"15min":50.137999999999998},"last":16.573},{"interface":"front","average":{"1min":17.808,"5min":17.616,"15min":16.946000000000002},"min":{"1min":10.286,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":51.164999999999999,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.154}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":18.66},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":17.364000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":13.91,"5min":15.550000000000001,"15min":18.908000000000001},"min":{"1min":7.71,"5min":7.71,"15min":7.71},"max":{"1min":24.242000000000001,"5min":48.231000000000002,"15min":104.834},"last":20.071000000000002},{"interface":"front","average":{"1min":17.329000000000001,"5min":19.853999999999999,"15min":20.48},"min":{"1min":11.135,"5min":11.135,"15min":11.135},"max":{"1min":25.175000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.388000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.974,"5min":16.959,"15min":18.948},"min":{"1min":8.1270000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":34.999000000000002,"5min":34.999000000000002,"15min":42.915999999999997},"last":10.621},{"interface":"front","average":{"1min":17.074000000000002,"5min":17.196999999999999,"15min":18.687000000000001},"min":{"1min":9.2289999999999992,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":33.512999999999998,"5min":43.697000000000003,"15min":43.697000000000003},"last":21.213000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":12.513,"5min":10.885,"15min":12.747999999999999},"min":{"1min":5.476,"5min":4.835,"15min":4.835},"max":{"1min":28.488,"5min":36.534999999999997,"15min":80.640000000000001},"last":23.114999999999998},{"interface":"front","average":{"1min":15.066000000000001,"5min":12.802,"15min":12.042},"min":{"1min":8.7309999999999999,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":26.504000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":25.140999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":16.111000000000001,"5min":14.246,"15min":18.893000000000001},"min":{"1min":10.234999999999999,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.82,"5min":32.081000000000003,"15min":100.703},"last":24.52},{"interface":"front","average":{"1min":9.2789999999999999,"5min":10.380000000000001,"15min":16.532},"min":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"max":{"1min":29.946999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.5919999999999996}]},{"osd":3,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.788,"5min":16.448,"15min":17.936},"min":{"1min":11.882999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":44.055,"5min":44.055,"15min":44.055},"last":26.606999999999999},{"interface":"front","average":{"1min":10.44,"5min":10.215,"15min":12.718999999999999},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":42.786000000000001,"5min":42.786000000000001,"15min":46.613},"last":6.0389999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.440000000000001,"5min":15.339,"15min":17.645},"min":{"1min":12.68,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":81.838999999999999},"last":27.620999999999999},{"interface":"front","average":{"1min":19.515999999999998,"5min":18.312999999999999,"15min":20.640999999999998},"min":{"1min":15.148,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":33.311,"5min":33.311,"15min":66.515000000000001},"last":28.974}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":9.8900000000000006},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":29.850999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":13.477,"5min":14.614000000000001,"15min":17.489999999999998},"min":{"1min":8.6739999999999995,"5min":8.2789999999999999,"15min":8.2789999999999999},"max":{"1min":19.747,"5min":31.954999999999998,"15min":58.619},"last":17.129999999999999},{"interface":"front","average":{"1min":12.161,"5min":14.904,"15min":19.318999999999999},"min":{"1min":5.8769999999999998,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":25.501999999999999,"5min":33.671999999999997,"15min":83.632000000000005},"last":21.196000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.297999999999998,"15min":21.789000000000001},"min":{"1min":12.069000000000001,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":34.576999999999998,"5min":34.576999999999998,"15min":62.006999999999998},"last":32.790999999999997},{"interface":"front","average":{"1min":13.202999999999999,"5min":15.308,"15min":19.100999999999999},"min":{"1min":8.9480000000000004,"5min":8.0719999999999992,"15min":8.0719999999999992},"max":{"1min":22.373000000000001,"5min":32.512999999999998,"15min":60.698999999999998},"last":19.228000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":10.68,"5min":10.488,"15min":17.312000000000001},"min":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":25.873000000000001,"5min":64.242999999999995,"15min":252.88399999999999},"last":8.1430000000000007},{"interface":"front","average":{"1min":12.551,"5min":11.323,"15min":18.364999999999998},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":33.127000000000002,"5min":63.192999999999998,"15min":274.91500000000002},"last":13.922000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.704000000000001,"5min":12.589,"15min":16.036999999999999},"min":{"1min":6.9960000000000004,"5min":5.093,"15min":5.093},"max":{"1min":23.831,"5min":35.383000000000003,"15min":82.866},"last":9.2059999999999995},{"interface":"front","average":{"1min":13.122,"5min":13.996,"15min":15.718},"min":{"1min":7.5750000000000002,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.742999999999999,"5min":34.289999999999999,"15min":85.991},"last":17.562999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.128,"5min":11.25,"15min":14.057},"min":{"1min":7.6230000000000002,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":24.594000000000001,"5min":41.045000000000002,"15min":76.593000000000004},"last":14.737},{"interface":"front","average":{"1min":16.071999999999999,"5min":15.656000000000001,"15min":19.260999999999999},"min":{"1min":10.727,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":26.637,"5min":42.284999999999997,"15min":85.162000000000006},"last":15.704000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":14.734,"5min":15.743,"15min":16.966000000000001},"min":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":24.876000000000001,"5min":76.387,"15min":76.387},"last":18.146999999999998},{"interface":"front","average":{"1min":13.502000000000001,"5min":13.249000000000001,"15min":13.815},"min":{"1min":7.7140000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":28.337,"5min":29.375,"15min":70.085999999999999},"last":16.645}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":19.116},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":12.494}]},{"osd":6,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":16.324999999999999,"15min":17.731000000000002},"min":{"1min":9.6999999999999993,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.141999999999999,"5min":31.550000000000001,"15min":75.471999999999994},"last":19.756},{"interface":"front","average":{"1min":18.029,"5min":17.558,"15min":22.172000000000001},"min":{"1min":13.818,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":30.916,"5min":37.479999999999997,"15min":84.617000000000004},"last":20.420999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":19.414999999999999,"5min":19.709,"15min":21.390000000000001},"min":{"1min":9.6750000000000007,"5min":9.6750000000000007,"15min":7.6219999999999999},"max":{"1min":48.314,"5min":78.506,"15min":84.031000000000006},"last":22.494},{"interface":"front","average":{"1min":19.506,"5min":17.164000000000001,"15min":18.596},"min":{"1min":12.209,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":49.362000000000002,"5min":49.362000000000002,"15min":77.974000000000004},"last":21.628}]}]},{"osd":4,"up_from":15,"seq":64424509533,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":10.154,"5min":8.6099999999999994,"15min":9.3480000000000008},"min":{"1min":4.4640000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":23.513000000000002,"5min":48.454000000000001,"15min":48.454000000000001},"last":14.603999999999999},{"interface":"front","average":{"1min":8.609,"5min":9.8610000000000007,"15min":10.148},"min":{"1min":5.0220000000000002,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":21.257000000000001,"5min":44.947000000000003,"15min":44.947000000000003},"last":4.5119999999999996}]},{"osd":1,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":10.202,"15min":11.420999999999999},"min":{"1min":5.9569999999999999,"5min":5.0090000000000003,"15min":5.0090000000000003},"max":{"1min":15.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":10.468},{"interface":"front","average":{"1min":12.343,"5min":12.006,"15min":13.369999999999999},"min":{"1min":7.149,"5min":6.5599999999999996,"15min":5.6269999999999998},"max":{"1min":25.908000000000001,"5min":26.003,"15min":31.027000000000001},"last":8.6440000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":13.532,"15min":12.522},"min":{"1min":7.9640000000000004,"5min":7.1379999999999999,"15min":7.1379999999999999},"max":{"1min":16.992000000000001,"5min":29.806999999999999,"15min":29.806999999999999},"last":19.222999999999999},{"interface":"front","average":{"1min":12.406000000000001,"5min":14.375,"15min":14.542},"min":{"1min":8.3629999999999995,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.27,"5min":32.915999999999997,"15min":32.915999999999997},"last":16.728999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:33 2023","interfaces":[{"interface":"back","average":{"1min":12.792999999999999,"5min":12.470000000000001,"15min":11.813000000000001},"min":{"1min":6.1180000000000003,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":24.111000000000001,"5min":28.991,"15min":28.991},"last":15.266999999999999},{"interface":"front","average":{"1min":15.817,"5min":15.462999999999999,"15min":14.531000000000001},"min":{"1min":9.1270000000000007,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":22.725000000000001,"5min":35.540999999999997,"15min":35.540999999999997},"last":11.093999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":13.359},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":7.891}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":17.841000000000001},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":19.827999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":13.946999999999999},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":18.460000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":12.375},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":7.1559999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":9.0570000000000004},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":6.2510000000000003}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":17.114999999999998},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":16.207999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.8539999999999992},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":15.446999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":37.387},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":38.234000000000002}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":14.472}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":11.664},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":13.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467975,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":1252,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.866}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.484000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.382999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.41}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.013999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.27}]}]},{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":18.798999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":15.212}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.4119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":12.298}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":13.695},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":18.061}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":15.82},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":20.184000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":19.553000000000001},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.619}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.649000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":14.57}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:38.962 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-06T08:04:38.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2023-12-06T08:04:39.708 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-06T08:04:39.718 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-06T08:04:39.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:40.344 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:40.344 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:40.356 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2023-12-06T08:04:39.876683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229068,"kb_used_data":7688,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745652,"statfs":{"total":773094113280,"available":772859547648,"internally_reserved":0,"allocated":7872512,"data_stored":3376936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.697665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.113198+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:31.113198+0000","last_peered":"2023-12-06T08:04:31.113198+0000","last_clean":"2023-12-06T08:04:31.113198+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:31.113198+0000","last_undegraded":"2023-12-06T08:04:31.113198+0000","last_fullsized":"2023-12-06T08:04:31.113198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:55:49.691370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.054362+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:30.054362+0000","last_peered":"2023-12-06T08:04:30.054362+0000","last_clean":"2023-12-06T08:04:30.054362+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:30.054362+0000","last_undegraded":"2023-12-06T08:04:30.054362+0000","last_fullsized":"2023-12-06T08:04:30.054362+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.110729+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:31.110729+0000","last_peered":"2023-12-06T08:04:31.110729+0000","last_clean":"2023-12-06T08:04:31.110729+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:31.110729+0000","last_undegraded":"2023-12-06T08:04:31.110729+0000","last_fullsized":"2023-12-06T08:04:31.110729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.196919+0000","last_change":"2023-12-06T08:04:12.508710+0000","last_active":"2023-12-06T08:04:31.196919+0000","last_peered":"2023-12-06T08:04:31.196919+0000","last_clean":"2023-12-06T08:04:31.196919+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:31.196919+0000","last_undegraded":"2023-12-06T08:04:31.196919+0000","last_fullsized":"2023-12-06T08:04:31.196919+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:22:19.396954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.100874+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.100874+0000","last_peered":"2023-12-06T08:04:31.100874+0000","last_clean":"2023-12-06T08:04:31.100874+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.100874+0000","last_undegraded":"2023-12-06T08:04:31.100874+0000","last_fullsized":"2023-12-06T08:04:31.100874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.050973+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:30.050973+0000","last_peered":"2023-12-06T08:04:30.050973+0000","last_clean":"2023-12-06T08:04:30.050973+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:30.050973+0000","last_undegraded":"2023-12-06T08:04:30.050973+0000","last_fullsized":"2023-12-06T08:04:30.050973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.152573+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:31.152573+0000","last_peered":"2023-12-06T08:04:31.152573+0000","last_clean":"2023-12-06T08:04:31.152573+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:31.152573+0000","last_undegraded":"2023-12-06T08:04:31.152573+0000","last_fullsized":"2023-12-06T08:04:31.152573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.005658+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:30.005658+0000","last_peered":"2023-12-06T08:04:30.005658+0000","last_clean":"2023-12-06T08:04:30.005658+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:30.005658+0000","last_undegraded":"2023-12-06T08:04:30.005658+0000","last_fullsized":"2023-12-06T08:04:30.005658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:01:11.918493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":234,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.108949+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:31.108949+0000","last_peered":"2023-12-06T08:04:31.108949+0000","last_clean":"2023-12-06T08:04:31.108949+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:31.108949+0000","last_undegraded":"2023-12-06T08:04:31.108949+0000","last_fullsized":"2023-12-06T08:04:31.108949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:55:38.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":17.556000000000001},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":11.894}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.2609999999999992},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":7.8019999999999996}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":12.472},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":7.3029999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":11.275},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.852}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":16.565999999999999},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.449}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.058},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":10.981}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":18.334},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":14.66}]}]},{"osd":2,"up_from":16,"seq":68719476828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":9.7089999999999996,"15min":10.234},"min":{"1min":4.9109999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":40.771999999999998,"5min":40.771999999999998,"15min":40.771999999999998},"last":12.353999999999999},{"interface":"front","average":{"1min":9.8580000000000005,"5min":9.8480000000000008,"15min":10.356},"min":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":45.781999999999996,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.665}]},{"osd":1,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.401,"5min":12.720000000000001,"15min":19.236000000000001},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":21.439,"5min":27.763000000000002,"15min":165.68600000000001},"last":13.706},{"interface":"front","average":{"1min":11.988,"5min":12.721,"15min":19.760999999999999},"min":{"1min":7.8300000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":17.602,"5min":27.974,"15min":153.68199999999999},"last":9.2010000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.388,"5min":10.52,"15min":11.021000000000001},"min":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":23.768999999999998,"5min":26.887,"15min":35.935000000000002},"last":15.625},{"interface":"front","average":{"1min":14.199999999999999,"5min":15.276,"15min":15.566000000000001},"min":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":25.352,"5min":30.433,"15min":30.433},"last":14.285}]},{"osd":4,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":13.138,"15min":15.73},"min":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":34.514000000000003,"5min":48.207999999999998,"15min":50.137999999999998},"last":16.573},{"interface":"front","average":{"1min":17.808,"5min":17.616,"15min":16.946000000000002},"min":{"1min":10.286,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":51.164999999999999,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.154}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":18.66},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":17.364000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":13.91,"5min":15.550000000000001,"15min":18.908000000000001},"min":{"1min":7.71,"5min":7.71,"15min":7.71},"max":{"1min":24.242000000000001,"5min":48.231000000000002,"15min":104.834},"last":20.071000000000002},{"interface":"front","average":{"1min":17.329000000000001,"5min":19.853999999999999,"15min":20.48},"min":{"1min":11.135,"5min":11.135,"15min":11.135},"max":{"1min":25.175000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.388000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.974,"5min":16.959,"15min":18.948},"min":{"1min":8.1270000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":34.999000000000002,"5min":34.999000000000002,"15min":42.915999999999997},"last":10.621},{"interface":"front","average":{"1min":17.074000000000002,"5min":17.196999999999999,"15min":18.687000000000001},"min":{"1min":9.2289999999999992,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":33.512999999999998,"5min":43.697000000000003,"15min":43.697000000000003},"last":21.213000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":12.513,"5min":10.885,"15min":12.747999999999999},"min":{"1min":5.476,"5min":4.835,"15min":4.835},"max":{"1min":28.488,"5min":36.534999999999997,"15min":80.640000000000001},"last":23.114999999999998},{"interface":"front","average":{"1min":15.066000000000001,"5min":12.802,"15min":12.042},"min":{"1min":8.7309999999999999,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":26.504000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":25.140999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":16.111000000000001,"5min":14.246,"15min":18.893000000000001},"min":{"1min":10.234999999999999,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.82,"5min":32.081000000000003,"15min":100.703},"last":24.52},{"interface":"front","average":{"1min":9.2789999999999999,"5min":10.380000000000001,"15min":16.532},"min":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"max":{"1min":29.946999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.5919999999999996}]},{"osd":3,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.788,"5min":16.448,"15min":17.936},"min":{"1min":11.882999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":44.055,"5min":44.055,"15min":44.055},"last":26.606999999999999},{"interface":"front","average":{"1min":10.44,"5min":10.215,"15min":12.718999999999999},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":42.786000000000001,"5min":42.786000000000001,"15min":46.613},"last":6.0389999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.440000000000001,"5min":15.339,"15min":17.645},"min":{"1min":12.68,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":81.838999999999999},"last":27.620999999999999},{"interface":"front","average":{"1min":19.515999999999998,"5min":18.312999999999999,"15min":20.640999999999998},"min":{"1min":15.148,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":33.311,"5min":33.311,"15min":66.515000000000001},"last":28.974}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":9.8900000000000006},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":29.850999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":13.477,"5min":14.614000000000001,"15min":17.489999999999998},"min":{"1min":8.6739999999999995,"5min":8.2789999999999999,"15min":8.2789999999999999},"max":{"1min":19.747,"5min":31.954999999999998,"15min":58.619},"last":17.129999999999999},{"interface":"front","average":{"1min":12.161,"5min":14.904,"15min":19.318999999999999},"min":{"1min":5.8769999999999998,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":25.501999999999999,"5min":33.671999999999997,"15min":83.632000000000005},"last":21.196000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.297999999999998,"15min":21.789000000000001},"min":{"1min":12.069000000000001,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":34.576999999999998,"5min":34.576999999999998,"15min":62.006999999999998},"last":32.790999999999997},{"interface":"front","average":{"1min":13.202999999999999,"5min":15.308,"15min":19.100999999999999},"min":{"1min":8.9480000000000004,"5min":8.0719999999999992,"15min":8.0719999999999992},"max":{"1min":22.373000000000001,"5min":32.512999999999998,"15min":60.698999999999998},"last":19.228000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":10.68,"5min":10.488,"15min":17.312000000000001},"min":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":25.873000000000001,"5min":64.242999999999995,"15min":252.88399999999999},"last":8.1430000000000007},{"interface":"front","average":{"1min":12.551,"5min":11.323,"15min":18.364999999999998},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":33.127000000000002,"5min":63.192999999999998,"15min":274.91500000000002},"last":13.922000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.704000000000001,"5min":12.589,"15min":16.036999999999999},"min":{"1min":6.9960000000000004,"5min":5.093,"15min":5.093},"max":{"1min":23.831,"5min":35.383000000000003,"15min":82.866},"last":9.2059999999999995},{"interface":"front","average":{"1min":13.122,"5min":13.996,"15min":15.718},"min":{"1min":7.5750000000000002,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.742999999999999,"5min":34.289999999999999,"15min":85.991},"last":17.562999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.128,"5min":11.25,"15min":14.057},"min":{"1min":7.6230000000000002,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":24.594000000000001,"5min":41.045000000000002,"15min":76.593000000000004},"last":14.737},{"interface":"front","average":{"1min":16.071999999999999,"5min":15.656000000000001,"15min":19.260999999999999},"min":{"1min":10.727,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":26.637,"5min":42.284999999999997,"15min":85.162000000000006},"last":15.704000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":14.734,"5min":15.743,"15min":16.966000000000001},"min":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":24.876000000000001,"5min":76.387,"15min":76.387},"last":18.146999999999998},{"interface":"front","average":{"1min":13.502000000000001,"5min":13.249000000000001,"15min":13.815},"min":{"1min":7.7140000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":28.337,"5min":29.375,"15min":70.085999999999999},"last":16.645}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":19.116},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":12.494}]},{"osd":6,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":16.324999999999999,"15min":17.731000000000002},"min":{"1min":9.6999999999999993,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.141999999999999,"5min":31.550000000000001,"15min":75.471999999999994},"last":19.756},{"interface":"front","average":{"1min":18.029,"5min":17.558,"15min":22.172000000000001},"min":{"1min":13.818,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":30.916,"5min":37.479999999999997,"15min":84.617000000000004},"last":20.420999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":19.414999999999999,"5min":19.709,"15min":21.390000000000001},"min":{"1min":9.6750000000000007,"5min":9.6750000000000007,"15min":7.6219999999999999},"max":{"1min":48.314,"5min":78.506,"15min":84.031000000000006},"last":22.494},{"interface":"front","average":{"1min":19.506,"5min":17.164000000000001,"15min":18.596},"min":{"1min":12.209,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":49.362000000000002,"5min":49.362000000000002,"15min":77.974000000000004},"last":21.628}]}]},{"osd":4,"up_from":15,"seq":64424509534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":8.984,"15min":9.3019999999999996},"min":{"1min":3.8500000000000001,"5min":3.8500000000000001,"15min":3.8500000000000001},"max":{"1min":15.817,"5min":48.454000000000001,"15min":48.454000000000001},"last":14.542},{"interface":"front","average":{"1min":9.9770000000000003,"5min":9.907,"15min":10.127000000000001},"min":{"1min":4.5119999999999996,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":18.184999999999999,"5min":44.947000000000003,"15min":44.947000000000003},"last":10.173999999999999}]},{"osd":1,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.3599999999999994,"5min":10.111000000000001,"15min":11.243},"min":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.0090000000000003},"max":{"1min":17.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":6.6230000000000002},{"interface":"front","average":{"1min":11.289,"5min":11.656000000000001,"15min":13.186},"min":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":5.6269999999999998},"max":{"1min":20.495000000000001,"5min":26.003,"15min":31.027000000000001},"last":10.75}]},{"osd":2,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":11.337,"5min":13.117000000000001,"15min":12.476000000000001},"min":{"1min":5.6440000000000001,"5min":5.6440000000000001,"15min":5.6440000000000001},"max":{"1min":19.222999999999999,"5min":29.806999999999999,"15min":29.806999999999999},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.704000000000001,"5min":14.004,"15min":14.481},"min":{"1min":7.2770000000000001,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.795999999999999,"5min":27.408000000000001,"15min":32.915999999999997},"last":11.385999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":12.711,"5min":12.853999999999999,"15min":11.895},"min":{"1min":6.4379999999999997,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.167000000000002,"5min":28.991,"15min":28.991},"last":25.167000000000002},{"interface":"front","average":{"1min":14.364000000000001,"5min":15.183,"15min":14.550000000000001},"min":{"1min":9.4640000000000004,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":20.163,"5min":35.540999999999997,"15min":35.540999999999997},"last":9.4640000000000004}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":23.859999999999999},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":15.962999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":22.946000000000002},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":29.797999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":12.108000000000001},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":28.934999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":12.375},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":7.1559999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":9.0570000000000004},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":6.2510000000000003}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":17.114999999999998},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":16.207999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.8539999999999992},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":15.446999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":37.387},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":38.234000000000002}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":14.472}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":11.664},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":13.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467976,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":1264,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8849999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.227}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.834}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.858000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.548999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":18.798999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":15.212}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.4119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":12.298}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":13.695},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":18.061}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":15.82},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":20.184000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":19.553000000000001},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.619}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.649000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":14.57}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:40.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:40.919 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:40.920 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:40.931 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2023-12-06T08:04:39.876683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229068,"kb_used_data":7688,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745652,"statfs":{"total":773094113280,"available":772859547648,"internally_reserved":0,"allocated":7872512,"data_stored":3376936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.697665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.113198+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:31.113198+0000","last_peered":"2023-12-06T08:04:31.113198+0000","last_clean":"2023-12-06T08:04:31.113198+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:31.113198+0000","last_undegraded":"2023-12-06T08:04:31.113198+0000","last_fullsized":"2023-12-06T08:04:31.113198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:55:49.691370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.054362+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:30.054362+0000","last_peered":"2023-12-06T08:04:30.054362+0000","last_clean":"2023-12-06T08:04:30.054362+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:30.054362+0000","last_undegraded":"2023-12-06T08:04:30.054362+0000","last_fullsized":"2023-12-06T08:04:30.054362+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.110729+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:31.110729+0000","last_peered":"2023-12-06T08:04:31.110729+0000","last_clean":"2023-12-06T08:04:31.110729+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:31.110729+0000","last_undegraded":"2023-12-06T08:04:31.110729+0000","last_fullsized":"2023-12-06T08:04:31.110729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.196919+0000","last_change":"2023-12-06T08:04:12.508710+0000","last_active":"2023-12-06T08:04:31.196919+0000","last_peered":"2023-12-06T08:04:31.196919+0000","last_clean":"2023-12-06T08:04:31.196919+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:31.196919+0000","last_undegraded":"2023-12-06T08:04:31.196919+0000","last_fullsized":"2023-12-06T08:04:31.196919+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:22:19.396954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.100874+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.100874+0000","last_peered":"2023-12-06T08:04:31.100874+0000","last_clean":"2023-12-06T08:04:31.100874+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.100874+0000","last_undegraded":"2023-12-06T08:04:31.100874+0000","last_fullsized":"2023-12-06T08:04:31.100874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.050973+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:30.050973+0000","last_peered":"2023-12-06T08:04:30.050973+0000","last_clean":"2023-12-06T08:04:30.050973+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:30.050973+0000","last_undegraded":"2023-12-06T08:04:30.050973+0000","last_fullsized":"2023-12-06T08:04:30.050973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.152573+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:31.152573+0000","last_peered":"2023-12-06T08:04:31.152573+0000","last_clean":"2023-12-06T08:04:31.152573+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:31.152573+0000","last_undegraded":"2023-12-06T08:04:31.152573+0000","last_fullsized":"2023-12-06T08:04:31.152573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.005658+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:30.005658+0000","last_peered":"2023-12-06T08:04:30.005658+0000","last_clean":"2023-12-06T08:04:30.005658+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:30.005658+0000","last_undegraded":"2023-12-06T08:04:30.005658+0000","last_fullsized":"2023-12-06T08:04:30.005658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:01:11.918493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":234,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.108949+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:31.108949+0000","last_peered":"2023-12-06T08:04:31.108949+0000","last_clean":"2023-12-06T08:04:31.108949+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:31.108949+0000","last_undegraded":"2023-12-06T08:04:31.108949+0000","last_fullsized":"2023-12-06T08:04:31.108949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:55:38.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":17.556000000000001},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":11.894}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.2609999999999992},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":7.8019999999999996}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":12.472},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":7.3029999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":11.275},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.852}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":16.565999999999999},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.449}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.058},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":10.981}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":18.334},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":14.66}]}]},{"osd":2,"up_from":16,"seq":68719476828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":9.7089999999999996,"15min":10.234},"min":{"1min":4.9109999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":40.771999999999998,"5min":40.771999999999998,"15min":40.771999999999998},"last":12.353999999999999},{"interface":"front","average":{"1min":9.8580000000000005,"5min":9.8480000000000008,"15min":10.356},"min":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":45.781999999999996,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.665}]},{"osd":1,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.401,"5min":12.720000000000001,"15min":19.236000000000001},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":21.439,"5min":27.763000000000002,"15min":165.68600000000001},"last":13.706},{"interface":"front","average":{"1min":11.988,"5min":12.721,"15min":19.760999999999999},"min":{"1min":7.8300000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":17.602,"5min":27.974,"15min":153.68199999999999},"last":9.2010000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.388,"5min":10.52,"15min":11.021000000000001},"min":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":23.768999999999998,"5min":26.887,"15min":35.935000000000002},"last":15.625},{"interface":"front","average":{"1min":14.199999999999999,"5min":15.276,"15min":15.566000000000001},"min":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":25.352,"5min":30.433,"15min":30.433},"last":14.285}]},{"osd":4,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":13.138,"15min":15.73},"min":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":34.514000000000003,"5min":48.207999999999998,"15min":50.137999999999998},"last":16.573},{"interface":"front","average":{"1min":17.808,"5min":17.616,"15min":16.946000000000002},"min":{"1min":10.286,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":51.164999999999999,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.154}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":18.66},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":17.364000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":13.91,"5min":15.550000000000001,"15min":18.908000000000001},"min":{"1min":7.71,"5min":7.71,"15min":7.71},"max":{"1min":24.242000000000001,"5min":48.231000000000002,"15min":104.834},"last":20.071000000000002},{"interface":"front","average":{"1min":17.329000000000001,"5min":19.853999999999999,"15min":20.48},"min":{"1min":11.135,"5min":11.135,"15min":11.135},"max":{"1min":25.175000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.388000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.974,"5min":16.959,"15min":18.948},"min":{"1min":8.1270000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":34.999000000000002,"5min":34.999000000000002,"15min":42.915999999999997},"last":10.621},{"interface":"front","average":{"1min":17.074000000000002,"5min":17.196999999999999,"15min":18.687000000000001},"min":{"1min":9.2289999999999992,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":33.512999999999998,"5min":43.697000000000003,"15min":43.697000000000003},"last":21.213000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":12.513,"5min":10.885,"15min":12.747999999999999},"min":{"1min":5.476,"5min":4.835,"15min":4.835},"max":{"1min":28.488,"5min":36.534999999999997,"15min":80.640000000000001},"last":23.114999999999998},{"interface":"front","average":{"1min":15.066000000000001,"5min":12.802,"15min":12.042},"min":{"1min":8.7309999999999999,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":26.504000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":25.140999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":16.111000000000001,"5min":14.246,"15min":18.893000000000001},"min":{"1min":10.234999999999999,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.82,"5min":32.081000000000003,"15min":100.703},"last":24.52},{"interface":"front","average":{"1min":9.2789999999999999,"5min":10.380000000000001,"15min":16.532},"min":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"max":{"1min":29.946999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.5919999999999996}]},{"osd":3,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.788,"5min":16.448,"15min":17.936},"min":{"1min":11.882999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":44.055,"5min":44.055,"15min":44.055},"last":26.606999999999999},{"interface":"front","average":{"1min":10.44,"5min":10.215,"15min":12.718999999999999},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":42.786000000000001,"5min":42.786000000000001,"15min":46.613},"last":6.0389999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.440000000000001,"5min":15.339,"15min":17.645},"min":{"1min":12.68,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":81.838999999999999},"last":27.620999999999999},{"interface":"front","average":{"1min":19.515999999999998,"5min":18.312999999999999,"15min":20.640999999999998},"min":{"1min":15.148,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":33.311,"5min":33.311,"15min":66.515000000000001},"last":28.974}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":9.8900000000000006},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":29.850999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":13.477,"5min":14.614000000000001,"15min":17.489999999999998},"min":{"1min":8.6739999999999995,"5min":8.2789999999999999,"15min":8.2789999999999999},"max":{"1min":19.747,"5min":31.954999999999998,"15min":58.619},"last":17.129999999999999},{"interface":"front","average":{"1min":12.161,"5min":14.904,"15min":19.318999999999999},"min":{"1min":5.8769999999999998,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":25.501999999999999,"5min":33.671999999999997,"15min":83.632000000000005},"last":21.196000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.297999999999998,"15min":21.789000000000001},"min":{"1min":12.069000000000001,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":34.576999999999998,"5min":34.576999999999998,"15min":62.006999999999998},"last":32.790999999999997},{"interface":"front","average":{"1min":13.202999999999999,"5min":15.308,"15min":19.100999999999999},"min":{"1min":8.9480000000000004,"5min":8.0719999999999992,"15min":8.0719999999999992},"max":{"1min":22.373000000000001,"5min":32.512999999999998,"15min":60.698999999999998},"last":19.228000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":10.68,"5min":10.488,"15min":17.312000000000001},"min":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":25.873000000000001,"5min":64.242999999999995,"15min":252.88399999999999},"last":8.1430000000000007},{"interface":"front","average":{"1min":12.551,"5min":11.323,"15min":18.364999999999998},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":33.127000000000002,"5min":63.192999999999998,"15min":274.91500000000002},"last":13.922000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.704000000000001,"5min":12.589,"15min":16.036999999999999},"min":{"1min":6.9960000000000004,"5min":5.093,"15min":5.093},"max":{"1min":23.831,"5min":35.383000000000003,"15min":82.866},"last":9.2059999999999995},{"interface":"front","average":{"1min":13.122,"5min":13.996,"15min":15.718},"min":{"1min":7.5750000000000002,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.742999999999999,"5min":34.289999999999999,"15min":85.991},"last":17.562999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.128,"5min":11.25,"15min":14.057},"min":{"1min":7.6230000000000002,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":24.594000000000001,"5min":41.045000000000002,"15min":76.593000000000004},"last":14.737},{"interface":"front","average":{"1min":16.071999999999999,"5min":15.656000000000001,"15min":19.260999999999999},"min":{"1min":10.727,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":26.637,"5min":42.284999999999997,"15min":85.162000000000006},"last":15.704000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":14.734,"5min":15.743,"15min":16.966000000000001},"min":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":24.876000000000001,"5min":76.387,"15min":76.387},"last":18.146999999999998},{"interface":"front","average":{"1min":13.502000000000001,"5min":13.249000000000001,"15min":13.815},"min":{"1min":7.7140000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":28.337,"5min":29.375,"15min":70.085999999999999},"last":16.645}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":19.116},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":12.494}]},{"osd":6,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":16.324999999999999,"15min":17.731000000000002},"min":{"1min":9.6999999999999993,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.141999999999999,"5min":31.550000000000001,"15min":75.471999999999994},"last":19.756},{"interface":"front","average":{"1min":18.029,"5min":17.558,"15min":22.172000000000001},"min":{"1min":13.818,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":30.916,"5min":37.479999999999997,"15min":84.617000000000004},"last":20.420999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":19.414999999999999,"5min":19.709,"15min":21.390000000000001},"min":{"1min":9.6750000000000007,"5min":9.6750000000000007,"15min":7.6219999999999999},"max":{"1min":48.314,"5min":78.506,"15min":84.031000000000006},"last":22.494},{"interface":"front","average":{"1min":19.506,"5min":17.164000000000001,"15min":18.596},"min":{"1min":12.209,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":49.362000000000002,"5min":49.362000000000002,"15min":77.974000000000004},"last":21.628}]}]},{"osd":4,"up_from":15,"seq":64424509534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":8.984,"15min":9.3019999999999996},"min":{"1min":3.8500000000000001,"5min":3.8500000000000001,"15min":3.8500000000000001},"max":{"1min":15.817,"5min":48.454000000000001,"15min":48.454000000000001},"last":14.542},{"interface":"front","average":{"1min":9.9770000000000003,"5min":9.907,"15min":10.127000000000001},"min":{"1min":4.5119999999999996,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":18.184999999999999,"5min":44.947000000000003,"15min":44.947000000000003},"last":10.173999999999999}]},{"osd":1,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.3599999999999994,"5min":10.111000000000001,"15min":11.243},"min":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.0090000000000003},"max":{"1min":17.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":6.6230000000000002},{"interface":"front","average":{"1min":11.289,"5min":11.656000000000001,"15min":13.186},"min":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":5.6269999999999998},"max":{"1min":20.495000000000001,"5min":26.003,"15min":31.027000000000001},"last":10.75}]},{"osd":2,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":11.337,"5min":13.117000000000001,"15min":12.476000000000001},"min":{"1min":5.6440000000000001,"5min":5.6440000000000001,"15min":5.6440000000000001},"max":{"1min":19.222999999999999,"5min":29.806999999999999,"15min":29.806999999999999},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.704000000000001,"5min":14.004,"15min":14.481},"min":{"1min":7.2770000000000001,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.795999999999999,"5min":27.408000000000001,"15min":32.915999999999997},"last":11.385999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":12.711,"5min":12.853999999999999,"15min":11.895},"min":{"1min":6.4379999999999997,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.167000000000002,"5min":28.991,"15min":28.991},"last":25.167000000000002},{"interface":"front","average":{"1min":14.364000000000001,"5min":15.183,"15min":14.550000000000001},"min":{"1min":9.4640000000000004,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":20.163,"5min":35.540999999999997,"15min":35.540999999999997},"last":9.4640000000000004}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":23.859999999999999},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":15.962999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":22.946000000000002},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":29.797999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":12.108000000000001},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":28.934999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":12.375},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":7.1559999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":9.0570000000000004},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":6.2510000000000003}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":17.114999999999998},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":16.207999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.8539999999999992},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":15.446999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":37.387},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":38.234000000000002}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":14.472}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":11.664},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":13.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467976,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":1264,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8849999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.227}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.834}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.858000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.548999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":18.798999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":15.212}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.4119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":12.298}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":13.695},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":18.061}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":15.82},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":20.184000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":19.553000000000001},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.619}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.649000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":14.57}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:40.933 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-06T08:04:40.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:41.489 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:41.502 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2023-12-06T08:04:39.876683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229068,"kb_used_data":7688,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745652,"statfs":{"total":773094113280,"available":772859547648,"internally_reserved":0,"allocated":7872512,"data_stored":3376936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.697665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.113198+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:31.113198+0000","last_peered":"2023-12-06T08:04:31.113198+0000","last_clean":"2023-12-06T08:04:31.113198+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:31.113198+0000","last_undegraded":"2023-12-06T08:04:31.113198+0000","last_fullsized":"2023-12-06T08:04:31.113198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:55:49.691370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.054362+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:30.054362+0000","last_peered":"2023-12-06T08:04:30.054362+0000","last_clean":"2023-12-06T08:04:30.054362+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:30.054362+0000","last_undegraded":"2023-12-06T08:04:30.054362+0000","last_fullsized":"2023-12-06T08:04:30.054362+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.110729+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:31.110729+0000","last_peered":"2023-12-06T08:04:31.110729+0000","last_clean":"2023-12-06T08:04:31.110729+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:31.110729+0000","last_undegraded":"2023-12-06T08:04:31.110729+0000","last_fullsized":"2023-12-06T08:04:31.110729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:16:08.484158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.196919+0000","last_change":"2023-12-06T08:04:12.508710+0000","last_active":"2023-12-06T08:04:31.196919+0000","last_peered":"2023-12-06T08:04:31.196919+0000","last_clean":"2023-12-06T08:04:31.196919+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:31.196919+0000","last_undegraded":"2023-12-06T08:04:31.196919+0000","last_fullsized":"2023-12-06T08:04:31.196919+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:22:19.396954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.100874+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.100874+0000","last_peered":"2023-12-06T08:04:31.100874+0000","last_clean":"2023-12-06T08:04:31.100874+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.100874+0000","last_undegraded":"2023-12-06T08:04:31.100874+0000","last_fullsized":"2023-12-06T08:04:31.100874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.050973+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:30.050973+0000","last_peered":"2023-12-06T08:04:30.050973+0000","last_clean":"2023-12-06T08:04:30.050973+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:30.050973+0000","last_undegraded":"2023-12-06T08:04:30.050973+0000","last_fullsized":"2023-12-06T08:04:30.050973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.152573+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:31.152573+0000","last_peered":"2023-12-06T08:04:31.152573+0000","last_clean":"2023-12-06T08:04:31.152573+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:31.152573+0000","last_undegraded":"2023-12-06T08:04:31.152573+0000","last_fullsized":"2023-12-06T08:04:31.152573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:50:36.974222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:30.005658+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:30.005658+0000","last_peered":"2023-12-06T08:04:30.005658+0000","last_clean":"2023-12-06T08:04:30.005658+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:30.005658+0000","last_undegraded":"2023-12-06T08:04:30.005658+0000","last_fullsized":"2023-12-06T08:04:30.005658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:01:11.918493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":234,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.108949+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:31.108949+0000","last_peered":"2023-12-06T08:04:31.108949+0000","last_clean":"2023-12-06T08:04:31.108949+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:31.108949+0000","last_undegraded":"2023-12-06T08:04:31.108949+0000","last_fullsized":"2023-12-06T08:04:31.108949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T14:55:38.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":17.556000000000001},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":11.894}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.2609999999999992},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":7.8019999999999996}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":12.472},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":7.3029999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":11.275},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.852}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":16.565999999999999},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.449}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.058},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":10.981}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":18.334},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":14.66}]}]},{"osd":2,"up_from":16,"seq":68719476828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":9.7089999999999996,"15min":10.234},"min":{"1min":4.9109999999999996,"5min":4.5359999999999996,"15min":4.5359999999999996},"max":{"1min":40.771999999999998,"5min":40.771999999999998,"15min":40.771999999999998},"last":12.353999999999999},{"interface":"front","average":{"1min":9.8580000000000005,"5min":9.8480000000000008,"15min":10.356},"min":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":45.781999999999996,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.665}]},{"osd":1,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.401,"5min":12.720000000000001,"15min":19.236000000000001},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":21.439,"5min":27.763000000000002,"15min":165.68600000000001},"last":13.706},{"interface":"front","average":{"1min":11.988,"5min":12.721,"15min":19.760999999999999},"min":{"1min":7.8300000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":17.602,"5min":27.974,"15min":153.68199999999999},"last":9.2010000000000005}]},{"osd":3,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":11.388,"5min":10.52,"15min":11.021000000000001},"min":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":23.768999999999998,"5min":26.887,"15min":35.935000000000002},"last":15.625},{"interface":"front","average":{"1min":14.199999999999999,"5min":15.276,"15min":15.566000000000001},"min":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":25.352,"5min":30.433,"15min":30.433},"last":14.285}]},{"osd":4,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.333,"5min":13.138,"15min":15.73},"min":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":34.514000000000003,"5min":48.207999999999998,"15min":50.137999999999998},"last":16.573},{"interface":"front","average":{"1min":17.808,"5min":17.616,"15min":16.946000000000002},"min":{"1min":10.286,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":51.164999999999999,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.154}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":18.66},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":17.364000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":13.91,"5min":15.550000000000001,"15min":18.908000000000001},"min":{"1min":7.71,"5min":7.71,"15min":7.71},"max":{"1min":24.242000000000001,"5min":48.231000000000002,"15min":104.834},"last":20.071000000000002},{"interface":"front","average":{"1min":17.329000000000001,"5min":19.853999999999999,"15min":20.48},"min":{"1min":11.135,"5min":11.135,"15min":11.135},"max":{"1min":25.175000000000001,"5min":49.146999999999998,"15min":49.146999999999998},"last":19.388000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:41 2023","interfaces":[{"interface":"back","average":{"1min":14.974,"5min":16.959,"15min":18.948},"min":{"1min":8.1270000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":34.999000000000002,"5min":34.999000000000002,"15min":42.915999999999997},"last":10.621},{"interface":"front","average":{"1min":17.074000000000002,"5min":17.196999999999999,"15min":18.687000000000001},"min":{"1min":9.2289999999999992,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":33.512999999999998,"5min":43.697000000000003,"15min":43.697000000000003},"last":21.213000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":719373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":12.513,"5min":10.885,"15min":12.747999999999999},"min":{"1min":5.476,"5min":4.835,"15min":4.835},"max":{"1min":28.488,"5min":36.534999999999997,"15min":80.640000000000001},"last":23.114999999999998},{"interface":"front","average":{"1min":15.066000000000001,"5min":12.802,"15min":12.042},"min":{"1min":8.7309999999999999,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":26.504000000000001,"5min":35.524000000000001,"15min":35.524000000000001},"last":25.140999999999998}]},{"osd":2,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":16.111000000000001,"5min":14.246,"15min":18.893000000000001},"min":{"1min":10.234999999999999,"5min":6.1280000000000001,"15min":6.1280000000000001},"max":{"1min":31.82,"5min":32.081000000000003,"15min":100.703},"last":24.52},{"interface":"front","average":{"1min":9.2789999999999999,"5min":10.380000000000001,"15min":16.532},"min":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"max":{"1min":29.946999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.5919999999999996}]},{"osd":3,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.788,"5min":16.448,"15min":17.936},"min":{"1min":11.882999999999999,"5min":8.0250000000000004,"15min":8.0250000000000004},"max":{"1min":44.055,"5min":44.055,"15min":44.055},"last":26.606999999999999},{"interface":"front","average":{"1min":10.44,"5min":10.215,"15min":12.718999999999999},"min":{"1min":5.657,"5min":5.657,"15min":5.657},"max":{"1min":42.786000000000001,"5min":42.786000000000001,"15min":46.613},"last":6.0389999999999997}]},{"osd":4,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":17.440000000000001,"5min":15.339,"15min":17.645},"min":{"1min":12.68,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":81.838999999999999},"last":27.620999999999999},{"interface":"front","average":{"1min":19.515999999999998,"5min":18.312999999999999,"15min":20.640999999999998},"min":{"1min":15.148,"5min":10.313000000000001,"15min":10.313000000000001},"max":{"1min":33.311,"5min":33.311,"15min":66.515000000000001},"last":28.974}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":9.8900000000000006},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":29.850999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":13.477,"5min":14.614000000000001,"15min":17.489999999999998},"min":{"1min":8.6739999999999995,"5min":8.2789999999999999,"15min":8.2789999999999999},"max":{"1min":19.747,"5min":31.954999999999998,"15min":58.619},"last":17.129999999999999},{"interface":"front","average":{"1min":12.161,"5min":14.904,"15min":19.318999999999999},"min":{"1min":5.8769999999999998,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":25.501999999999999,"5min":33.671999999999997,"15min":83.632000000000005},"last":21.196000000000002}]},{"osd":7,"last update":"Wed Dec 6 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":19.152999999999999,"5min":19.297999999999998,"15min":21.789000000000001},"min":{"1min":12.069000000000001,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":34.576999999999998,"5min":34.576999999999998,"15min":62.006999999999998},"last":32.790999999999997},{"interface":"front","average":{"1min":13.202999999999999,"5min":15.308,"15min":19.100999999999999},"min":{"1min":8.9480000000000004,"5min":8.0719999999999992,"15min":8.0719999999999992},"max":{"1min":22.373000000000001,"5min":32.512999999999998,"15min":60.698999999999998},"last":19.228000000000002}]}]},{"osd":0,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":10.68,"5min":10.488,"15min":17.312000000000001},"min":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":25.873000000000001,"5min":64.242999999999995,"15min":252.88399999999999},"last":8.1430000000000007},{"interface":"front","average":{"1min":12.551,"5min":11.323,"15min":18.364999999999998},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":33.127000000000002,"5min":63.192999999999998,"15min":274.91500000000002},"last":13.922000000000001}]},{"osd":2,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.704000000000001,"5min":12.589,"15min":16.036999999999999},"min":{"1min":6.9960000000000004,"5min":5.093,"15min":5.093},"max":{"1min":23.831,"5min":35.383000000000003,"15min":82.866},"last":9.2059999999999995},{"interface":"front","average":{"1min":13.122,"5min":13.996,"15min":15.718},"min":{"1min":7.5750000000000002,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.742999999999999,"5min":34.289999999999999,"15min":85.991},"last":17.562999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":12.128,"5min":11.25,"15min":14.057},"min":{"1min":7.6230000000000002,"5min":4.5970000000000004,"15min":4.5970000000000004},"max":{"1min":24.594000000000001,"5min":41.045000000000002,"15min":76.593000000000004},"last":14.737},{"interface":"front","average":{"1min":16.071999999999999,"5min":15.656000000000001,"15min":19.260999999999999},"min":{"1min":10.727,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":26.637,"5min":42.284999999999997,"15min":85.162000000000006},"last":15.704000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":14.734,"5min":15.743,"15min":16.966000000000001},"min":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":24.876000000000001,"5min":76.387,"15min":76.387},"last":18.146999999999998},{"interface":"front","average":{"1min":13.502000000000001,"5min":13.249000000000001,"15min":13.815},"min":{"1min":7.7140000000000004,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":28.337,"5min":29.375,"15min":70.085999999999999},"last":16.645}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":19.116},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":12.494}]},{"osd":6,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":16.324999999999999,"15min":17.731000000000002},"min":{"1min":9.6999999999999993,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":30.141999999999999,"5min":31.550000000000001,"15min":75.471999999999994},"last":19.756},{"interface":"front","average":{"1min":18.029,"5min":17.558,"15min":22.172000000000001},"min":{"1min":13.818,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":30.916,"5min":37.479999999999997,"15min":84.617000000000004},"last":20.420999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:03:36 2023","interfaces":[{"interface":"back","average":{"1min":19.414999999999999,"5min":19.709,"15min":21.390000000000001},"min":{"1min":9.6750000000000007,"5min":9.6750000000000007,"15min":7.6219999999999999},"max":{"1min":48.314,"5min":78.506,"15min":84.031000000000006},"last":22.494},{"interface":"front","average":{"1min":19.506,"5min":17.164000000000001,"15min":18.596},"min":{"1min":12.209,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":49.362000000000002,"5min":49.362000000000002,"15min":77.974000000000004},"last":21.628}]}]},{"osd":4,"up_from":15,"seq":64424509534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":8.984,"15min":9.3019999999999996},"min":{"1min":3.8500000000000001,"5min":3.8500000000000001,"15min":3.8500000000000001},"max":{"1min":15.817,"5min":48.454000000000001,"15min":48.454000000000001},"last":14.542},{"interface":"front","average":{"1min":9.9770000000000003,"5min":9.907,"15min":10.127000000000001},"min":{"1min":4.5119999999999996,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":18.184999999999999,"5min":44.947000000000003,"15min":44.947000000000003},"last":10.173999999999999}]},{"osd":1,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.3599999999999994,"5min":10.111000000000001,"15min":11.243},"min":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.0090000000000003},"max":{"1min":17.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":6.6230000000000002},{"interface":"front","average":{"1min":11.289,"5min":11.656000000000001,"15min":13.186},"min":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":5.6269999999999998},"max":{"1min":20.495000000000001,"5min":26.003,"15min":31.027000000000001},"last":10.75}]},{"osd":2,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":11.337,"5min":13.117000000000001,"15min":12.476000000000001},"min":{"1min":5.6440000000000001,"5min":5.6440000000000001,"15min":5.6440000000000001},"max":{"1min":19.222999999999999,"5min":29.806999999999999,"15min":29.806999999999999},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.704000000000001,"5min":14.004,"15min":14.481},"min":{"1min":7.2770000000000001,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.795999999999999,"5min":27.408000000000001,"15min":32.915999999999997},"last":11.385999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":12.711,"5min":12.853999999999999,"15min":11.895},"min":{"1min":6.4379999999999997,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.167000000000002,"5min":28.991,"15min":28.991},"last":25.167000000000002},{"interface":"front","average":{"1min":14.364000000000001,"5min":15.183,"15min":14.550000000000001},"min":{"1min":9.4640000000000004,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":20.163,"5min":35.540999999999997,"15min":35.540999999999997},"last":9.4640000000000004}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":23.859999999999999},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":15.962999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":22.946000000000002},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":29.797999999999998}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":12.108000000000001},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":28.934999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":12.375},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":7.1559999999999997}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":9.0570000000000004},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":6.2510000000000003}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":17.114999999999998},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":16.207999999999998}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.8539999999999992},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":15.446999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":37.387},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":38.234000000000002}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":14.472}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":11.664},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":13.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467976,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":1264,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8849999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.227}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.834}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.858000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.548999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":321533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":18.798999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":15.212}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":7.4119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":12.816000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":12.298}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":13.695},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":18.061}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":15.82},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":20.184000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":19.553000000000001},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":16.619}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.649000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":14.57}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:41.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-06T08:04:42.179 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:42.179 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":127,"fsid":"aba07bce-2203-46e5-bd3f-65b300a2f9db","created":"2023-12-06T07:56:32.091884+0000","modified":"2023-12-06T08:04:34.982664+0000","last_up_change":"2023-12-06T08:04:10.917942+0000","last_in_change":"2023-12-06T08:04:10.917942+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-06T07:57:18.316796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T07:57:26.381263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0da60772-f74a-424c-b41a-462ed8a484ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1405545511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":1405545511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":1405545511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1405545511}]},"public_addr":"172.21.15.84:6800/1405545511","cluster_addr":"172.21.15.84:6801/1405545511","heartbeat_back_addr":"172.21.15.84:6803/1405545511","heartbeat_front_addr":"172.21.15.84:6802/1405545511","state":["exists","up"]},{"osd":1,"uuid":"aa3bcc32-a99b-4382-b6d8-a9561dc53073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1973529083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":1973529083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":1973529083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1973529083}]},"public_addr":"172.21.15.84:6812/1973529083","cluster_addr":"172.21.15.84:6813/1973529083","heartbeat_back_addr":"172.21.15.84:6815/1973529083","heartbeat_front_addr":"172.21.15.84:6814/1973529083","state":["exists","up"]},{"osd":2,"uuid":"ea5cc880-085f-4d6a-9307-212ab3aee1ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1651824594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":1651824594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":1651824594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1651824594}]},"public_addr":"172.21.15.84:6804/1651824594","cluster_addr":"172.21.15.84:6805/1651824594","heartbeat_back_addr":"172.21.15.84:6807/1651824594","heartbeat_front_addr":"172.21.15.84:6806/1651824594","state":["exists","up"]},{"osd":3,"uuid":"f3f7cd68-c256-4e03-9a53-5a88e9e19c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3951389031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3951389031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3951389031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3951389031}]},"public_addr":"172.21.15.84:6808/3951389031","cluster_addr":"172.21.15.84:6809/3951389031","heartbeat_back_addr":"172.21.15.84:6811/3951389031","heartbeat_front_addr":"172.21.15.84:6810/3951389031","state":["exists","up"]},{"osd":4,"uuid":"d078280f-f9dc-490e-a13f-4714b9d70ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":2922507264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2922507264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2922507264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":2922507264}]},"public_addr":"172.21.15.107:6805/2922507264","cluster_addr":"172.21.15.107:6806/2922507264","heartbeat_back_addr":"172.21.15.107:6808/2922507264","heartbeat_front_addr":"172.21.15.107:6807/2922507264","state":["exists","up"]},{"osd":5,"uuid":"03c3b27d-6041-413a-a1d5-807f1349decb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":93,"up_from":108,"up_thru":108,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3236700828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3236700828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3236700828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":3236700828}]},"public_addr":"172.21.15.107:6809/3236700828","cluster_addr":"172.21.15.107:6810/3236700828","heartbeat_back_addr":"172.21.15.107:6812/3236700828","heartbeat_front_addr":"172.21.15.107:6811/3236700828","state":["exists","up"]},{"osd":6,"uuid":"332ffee8-2eca-481a-a2a4-ca5edfbebfe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":584359350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":584359350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":584359350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6815","nonce":584359350}]},"public_addr":"172.21.15.107:6813/584359350","cluster_addr":"172.21.15.107:6814/584359350","heartbeat_back_addr":"172.21.15.107:6816/584359350","heartbeat_front_addr":"172.21.15.107:6815/584359350","state":["exists","up"]},{"osd":7,"uuid":"6eab830f-d790-454c-9c56-5092fab218c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":1873215283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1873215283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1873215283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":1873215283}]},"public_addr":"172.21.15.107:6801/1873215283","cluster_addr":"172.21.15.107:6802/1873215283","heartbeat_back_addr":"172.21.15.107:6804/1873215283","heartbeat_front_addr":"172.21.15.107:6803/1873215283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.731245+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.384135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.655702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:15.515341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.584483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.330101+0000","dead_epoch":93},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:14.433563+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T07:57:13.629220+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[4,3]},{"pgid":"2.3","osds":[0,3]},{"pgid":"2.5","osds":[1,6]},{"pgid":"2.6","osds":[7,2]},{"pgid":"2.7","osds":[1,7]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T08:04:43.193 INFO:tasks.ceph:Scrubbing osd.0 2023-12-06T08:04:43.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:43.384 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:43.384 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:43.384 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:43.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-06T08:04:43.953 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 0 to deep-scrub 2023-12-06T08:04:43.966 INFO:tasks.ceph:Scrubbing osd.1 2023-12-06T08:04:43.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:44.157 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:44.158 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:44.158 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:44.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-06T08:04:44.753 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 1 to deep-scrub 2023-12-06T08:04:44.767 INFO:tasks.ceph:Scrubbing osd.2 2023-12-06T08:04:44.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:44.963 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:44.964 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:44.964 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:44.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-06T08:04:45.113 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:41.545 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.116 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:41.548 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.141 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:03:41.574 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.163 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:39.117 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.164 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:26.827 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.165 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:39.119 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.166 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:26.830 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.186 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:39.138 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.187 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:26.850 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:04:45.338 INFO:teuthology.orchestra.run.smithi084.stderr:2023-12-06T08:04:45.338+0000 7feecbfff640 0 --2- 172.21.15.84:0/4158458936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feecc0720f0 0x7feecc13eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-06T08:04:45.817 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 2 to deep-scrub 2023-12-06T08:04:45.831 INFO:tasks.ceph:Scrubbing osd.3 2023-12-06T08:04:45.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:46.058 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:46.058 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:46.058 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:46.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-06T08:04:46.635 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 3 to deep-scrub 2023-12-06T08:04:46.648 INFO:tasks.ceph:Scrubbing osd.4 2023-12-06T08:04:46.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:46.888 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:46.889 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:46.889 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:46.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-06T08:04:47.435 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 4 to deep-scrub 2023-12-06T08:04:47.449 INFO:tasks.ceph:Scrubbing osd.5 2023-12-06T08:04:47.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:47.650 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:47.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-06T08:04:48.244 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 5 to deep-scrub 2023-12-06T08:04:48.257 INFO:tasks.ceph:Scrubbing osd.6 2023-12-06T08:04:48.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:48.446 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:48.446 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:48.447 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:48.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-06T08:04:49.035 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 6 to deep-scrub 2023-12-06T08:04:49.049 INFO:tasks.ceph:Scrubbing osd.7 2023-12-06T08:04:49.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-06T08:04:49.237 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2023-12-06T08:04:49.237 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-06T08:04:49.237 INFO:teuthology.orchestra.run.smithi084.stdout:} 2023-12-06T08:04:49.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-06T08:04:49.794 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 7 to deep-scrub 2023-12-06T08:04:49.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:04:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:04:50.367 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:04:50.379 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2023-12-06T08:04:49.878694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229100,"kb_used_data":7760,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745620,"statfs":{"total":773094113280,"available":772859514880,"internally_reserved":0,"allocated":7946240,"data_stored":3403912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002308"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:35.823894+0000","last_change":"2023-12-06T08:04:12.353504+0000","last_active":"2023-12-06T08:04:35.823894+0000","last_peered":"2023-12-06T08:04:35.823894+0000","last_clean":"2023-12-06T08:04:35.823894+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:35.823894+0000","last_undegraded":"2023-12-06T08:04:35.823894+0000","last_fullsized":"2023-12-06T08:04:35.823894+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.639474+0000","last_change":"2023-12-06T08:04:12.351413+0000","last_active":"2023-12-06T08:04:31.639474+0000","last_peered":"2023-12-06T08:04:31.639474+0000","last_clean":"2023-12-06T08:04:31.639474+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:31.639474+0000","last_undegraded":"2023-12-06T08:04:31.639474+0000","last_fullsized":"2023-12-06T08:04:31.639474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:10:05.136583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":107,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:35.820098+0000","last_change":"2023-12-06T07:58:33.855956+0000","last_active":"2023-12-06T08:04:35.820098+0000","last_peered":"2023-12-06T08:04:35.820098+0000","last_clean":"2023-12-06T08:04:35.820098+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:35.820098+0000","last_undegraded":"2023-12-06T08:04:35.820098+0000","last_fullsized":"2023-12-06T08:04:35.820098+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-06T08:04:49.382742+0000","last_change":"2023-12-06T08:04:49.274266+0000","last_active":"2023-12-06T08:04:49.382742+0000","last_peered":"2023-12-06T08:04:49.382742+0000","last_clean":"2023-12-06T08:04:49.382742+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:49.382742+0000","last_undegraded":"2023-12-06T08:04:49.382742+0000","last_fullsized":"2023-12-06T08:04:49.382742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":127,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.650226+0000","last_change":"2023-12-06T08:02:14.384066+0000","last_active":"2023-12-06T08:04:31.650226+0000","last_peered":"2023-12-06T08:04:31.650226+0000","last_clean":"2023-12-06T08:04:31.650226+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:31.650226+0000","last_undegraded":"2023-12-06T08:04:31.650226+0000","last_fullsized":"2023-12-06T08:04:31.650226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T15:11:49.283842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-12-06T08:04:31.637622+0000","last_change":"2023-12-06T08:04:12.544698+0000","last_active":"2023-12-06T08:04:31.637622+0000","last_peered":"2023-12-06T08:04:31.637622+0000","last_clean":"2023-12-06T08:04:31.637622+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:31.637622+0000","last_undegraded":"2023-12-06T08:04:31.637622+0000","last_fullsized":"2023-12-06T08:04:31.637622+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:08:53.677006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:36.949504+0000","last_change":"2023-12-06T08:00:11.358713+0000","last_active":"2023-12-06T08:04:36.949504+0000","last_peered":"2023-12-06T08:04:36.949504+0000","last_clean":"2023-12-06T08:04:36.949504+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:36.949504+0000","last_undegraded":"2023-12-06T08:04:36.949504+0000","last_fullsized":"2023-12-06T08:04:36.949504+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":84,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:37.518751+0000","last_change":"2023-12-06T08:04:16.726389+0000","last_active":"2023-12-06T08:04:37.518751+0000","last_peered":"2023-12-06T08:04:37.518751+0000","last_clean":"2023-12-06T08:04:37.518751+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:37.518751+0000","last_undegraded":"2023-12-06T08:04:37.518751+0000","last_fullsized":"2023-12-06T08:04:37.518751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:27.431447+0000","last_clean_scrub_stamp":"2023-12-06T07:57:27.431447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":236,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:35.816897+0000","last_change":"2023-12-06T08:04:12.557988+0000","last_active":"2023-12-06T08:04:35.816897+0000","last_peered":"2023-12-06T08:04:35.816897+0000","last_clean":"2023-12-06T08:04:35.816897+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:35.816897+0000","last_undegraded":"2023-12-06T08:04:35.816897+0000","last_fullsized":"2023-12-06T08:04:35.816897+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T07:57:18.326272+0000","last_clean_scrub_stamp":"2023-12-06T07:57:18.326272+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":8.6609999999999996},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":9.125}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":9.8320000000000007},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":10.699999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":16.350999999999999},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":12.617000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":14.741},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":17.754000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":19.286000000000001},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":12.052}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":16.948},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":13.865}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":15.429},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":18.664000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":9.8439999999999994,"15min":10.177},"min":{"1min":4.2050000000000001,"5min":4.2050000000000001,"15min":4.2050000000000001},"max":{"1min":22.251999999999999,"5min":40.771999999999998,"15min":40.771999999999998},"last":6.2229999999999999},{"interface":"front","average":{"1min":8.4399999999999995,"5min":9.3480000000000008,"15min":10.211},"min":{"1min":5.673,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":14.021000000000001,"5min":45.781999999999996,"15min":45.781999999999996},"last":7.7069999999999999}]},{"osd":1,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.792999999999999,"5min":12.814,"15min":18.655999999999999},"min":{"1min":6.8959999999999999,"5min":5.657,"15min":5.657},"max":{"1min":39.911999999999999,"5min":39.911999999999999,"15min":165.68600000000001},"last":12.743},{"interface":"front","average":{"1min":13.313000000000001,"5min":13.093,"15min":19.096},"min":{"1min":5.923,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":30.786999999999999,"5min":30.786999999999999,"15min":153.68199999999999},"last":10.029999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.664999999999999,"5min":11.467000000000001,"15min":11.18},"min":{"1min":7.0490000000000004,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":40.226999999999997,"5min":40.226999999999997,"15min":40.226999999999997},"last":10.965},{"interface":"front","average":{"1min":16.241,"5min":15.218,"15min":15.601000000000001},"min":{"1min":9.0559999999999992,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":36.771999999999998,"5min":36.771999999999998,"15min":36.771999999999998},"last":13.396000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.721,"5min":13.516,"15min":15.509},"min":{"1min":8.048,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":23.898,"5min":48.207999999999998,"15min":50.137999999999998},"last":12.186},{"interface":"front","average":{"1min":17.294,"5min":17.658000000000001,"15min":16.991},"min":{"1min":9.8680000000000003,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":29.786000000000001,"5min":51.164999999999999,"15min":51.164999999999999},"last":14.025}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":16.838000000000001},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":18.707000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.763999999999999,"15min":18.670999999999999},"min":{"1min":8.8010000000000002,"5min":7.71,"15min":7.71},"max":{"1min":31.346,"5min":44.640999999999998,"15min":104.834},"last":11.622},{"interface":"front","average":{"1min":20.312999999999999,"5min":19.507999999999999,"15min":20.448},"min":{"1min":12.666,"5min":11.135,"15min":11.135},"max":{"1min":33.591000000000001,"5min":43.287999999999997,"15min":49.146999999999998},"last":19.495999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":16.402000000000001,"5min":16.791,"15min":18.712},"min":{"1min":8.3320000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":35.655000000000001,"5min":35.655000000000001,"15min":42.915999999999997},"last":17.516999999999999},{"interface":"front","average":{"1min":18.847999999999999,"5min":17.591000000000001,"15min":18.648},"min":{"1min":10.771000000000001,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":35.119999999999997,"5min":43.697000000000003,"15min":43.697000000000003},"last":15.013999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":14.696999999999999,"5min":11.881,"15min":12.816000000000001},"min":{"1min":5.2190000000000003,"5min":4.835,"15min":4.835},"max":{"1min":86.849999999999994,"5min":86.849999999999994,"15min":86.849999999999994},"last":9.6969999999999992},{"interface":"front","average":{"1min":16.329999999999998,"5min":14.273,"15min":12.353},"min":{"1min":8.5570000000000004,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":76.355000000000004,"5min":76.355000000000004,"15min":76.355000000000004},"last":13.327999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":14.247,"15min":18.376999999999999},"min":{"1min":8.7599999999999998,"5min":7.0940000000000003,"15min":6.1280000000000001},"max":{"1min":24.52,"5min":32.081000000000003,"15min":100.703},"last":13.992000000000001},{"interface":"front","average":{"1min":8.0730000000000004,"5min":9.3610000000000007,"15min":15.763},"min":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":17.173999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":7.6600000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":16.475999999999999,"15min":17.669},"min":{"1min":9.9329999999999998,"5min":9.9329999999999998,"15min":8.0250000000000004},"max":{"1min":26.606999999999999,"5min":44.055,"15min":44.055},"last":14.606},{"interface":"front","average":{"1min":8.3390000000000004,"5min":9.9789999999999992,"15min":12.343},"min":{"1min":5.5449999999999999,"5min":5.5449999999999999,"15min":5.5449999999999999},"max":{"1min":23.137,"5min":42.786000000000001,"15min":46.613},"last":6.359}]},{"osd":4,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":15.102,"5min":15.279999999999999,"15min":17.399000000000001},"min":{"1min":10.817,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":27.620999999999999,"5min":32.173000000000002,"15min":81.838999999999999},"last":15.878},{"interface":"front","average":{"1min":17.588000000000001,"5min":18.548999999999999,"15min":20.359999999999999},"min":{"1min":13.970000000000001,"5min":10.754,"15min":10.313000000000001},"max":{"1min":28.974,"5min":33.311,"15min":66.515000000000001},"last":20.274999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":12.456},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":15.308999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":11.569000000000001,"5min":14.236000000000001,"15min":17},"min":{"1min":6.4820000000000002,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.640000000000001,"5min":31.954999999999998,"15min":58.619},"last":7.0830000000000002},{"interface":"front","average":{"1min":10.394,"5min":13.231999999999999,"15min":18.577000000000002},"min":{"1min":5.8780000000000001,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":21.196000000000002,"5min":33.671999999999997,"15min":83.632000000000005},"last":18.251999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":18.204999999999998,"5min":19.140999999999998,"15min":21.466999999999999},"min":{"1min":12.698,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":32.790999999999997,"5min":34.576999999999998,"15min":62.006999999999998},"last":19.303999999999998},{"interface":"front","average":{"1min":11.766,"5min":14.564,"15min":18.484999999999999},"min":{"1min":7.4500000000000002,"5min":7.4500000000000002,"15min":7.4500000000000002},"max":{"1min":32.295000000000002,"5min":32.512999999999998,"15min":60.698999999999998},"last":10.768000000000001}]}]},{"osd":0,"up_from":16,"seq":68719476832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":7.9240000000000004,"5min":10.243,"15min":16.459},"min":{"1min":4.6879999999999997,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":12.43,"5min":64.242999999999995,"15min":252.88399999999999},"last":5.0010000000000003},{"interface":"front","average":{"1min":9.8780000000000001,"5min":11.519,"15min":17.565000000000001},"min":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"max":{"1min":19.867000000000001,"5min":63.192999999999998,"15min":274.91500000000002},"last":6.9119999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":12.132,"5min":12.692,"15min":15.661},"min":{"1min":7.5030000000000001,"5min":5.093,"15min":5.093},"max":{"1min":24.282,"5min":35.383000000000003,"15min":82.866},"last":8.2010000000000005},{"interface":"front","average":{"1min":13.413,"5min":13.98,"15min":15.507},"min":{"1min":7.8280000000000003,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.916,"5min":34.289999999999999,"15min":85.991},"last":8.8369999999999997}]},{"osd":3,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":8.8049999999999997,"5min":10.826000000000001,"15min":13.613},"min":{"1min":5.984,"5min":5.984,"15min":4.5970000000000004},"max":{"1min":14.737,"5min":41.045000000000002,"15min":76.593000000000004},"last":12.807},{"interface":"front","average":{"1min":13.013999999999999,"5min":15.478,"15min":18.724},"min":{"1min":8.4619999999999997,"5min":7.5890000000000004,"15min":6.9960000000000004},"max":{"1min":16.349,"5min":42.284999999999997,"15min":85.162000000000006},"last":12.105}]},{"osd":4,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":14.909000000000001,"5min":15.869999999999999,"15min":16.788},"min":{"1min":8.9260000000000002,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":26.155000000000001,"5min":76.387,"15min":76.387},"last":9.4580000000000002},{"interface":"front","average":{"1min":11.792999999999999,"5min":13.420999999999999,"15min":13.662000000000001},"min":{"1min":6.9669999999999996,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":18.969000000000001,"5min":29.375,"15min":70.085999999999999},"last":14.880000000000001}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":15.816000000000001},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":14.128}]},{"osd":6,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":14.260999999999999,"5min":16.559999999999999,"15min":17.452999999999999},"min":{"1min":9.7309999999999999,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":19.756,"5min":31.550000000000001,"15min":75.471999999999994},"last":16.574999999999999},{"interface":"front","average":{"1min":16.353000000000002,"5min":17.459,"15min":21.631},"min":{"1min":10.074999999999999,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":27.234999999999999,"5min":37.479999999999997,"15min":84.617000000000004},"last":10.574999999999999}]},{"osd":7,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":15.933,"5min":19.382000000000001,"15min":20.969999999999999},"min":{"1min":8.9049999999999994,"5min":8.9049999999999994,"15min":7.6219999999999999},"max":{"1min":28.501000000000001,"5min":78.506,"15min":84.031000000000006},"last":11.343999999999999},{"interface":"front","average":{"1min":16.109000000000002,"5min":17.524000000000001,"15min":18.382999999999999},"min":{"1min":9.7370000000000001,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":21.628,"5min":49.362000000000002,"15min":77.974000000000004},"last":17.382999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":8.984,"15min":9.3019999999999996},"min":{"1min":3.8500000000000001,"5min":3.8500000000000001,"15min":3.8500000000000001},"max":{"1min":15.817,"5min":48.454000000000001,"15min":48.454000000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":9.9770000000000003,"5min":9.907,"15min":10.127000000000001},"min":{"1min":4.5119999999999996,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":18.184999999999999,"5min":44.947000000000003,"15min":44.947000000000003},"last":6.5650000000000004}]},{"osd":1,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.3599999999999994,"5min":10.111000000000001,"15min":11.243},"min":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.0090000000000003},"max":{"1min":17.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":7.835},{"interface":"front","average":{"1min":11.289,"5min":11.656000000000001,"15min":13.186},"min":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":5.6269999999999998},"max":{"1min":20.495000000000001,"5min":26.003,"15min":31.027000000000001},"last":9.6579999999999995}]},{"osd":2,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":11.337,"5min":13.117000000000001,"15min":12.476000000000001},"min":{"1min":5.6440000000000001,"5min":5.6440000000000001,"15min":5.6440000000000001},"max":{"1min":19.222999999999999,"5min":29.806999999999999,"15min":29.806999999999999},"last":12.965999999999999},{"interface":"front","average":{"1min":13.704000000000001,"5min":14.004,"15min":14.481},"min":{"1min":7.2770000000000001,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.795999999999999,"5min":27.408000000000001,"15min":32.915999999999997},"last":15.878}]},{"osd":3,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":12.711,"5min":12.853999999999999,"15min":11.895},"min":{"1min":6.4379999999999997,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.167000000000002,"5min":28.991,"15min":28.991},"last":10.433999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":15.183,"15min":14.550000000000001},"min":{"1min":9.4640000000000004,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":20.163,"5min":35.540999999999997,"15min":35.540999999999997},"last":13.558999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":14.789999999999999},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":17.341999999999999}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":11.664999999999999},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":15.358000000000001}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":16.530000000000001},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":14.199}]}]},{"osd":6,"up_from":14,"seq":60129542238,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":18.805},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":12.096}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":10.397},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":8.6799999999999997}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":11.065},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":13.907}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":15.012},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":16.652999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":12.648999999999999},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":13.291}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":17.257000000000001},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":18.081}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":19.588000000000001},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":14.452999999999999}]}]},{"osd":5,"up_from":108,"seq":463856467978,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":1264,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7439999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4489999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.212}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.973000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.324999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.667000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":17.699999999999999},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":18.271000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":7.7439999999999998},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":16.113}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":15.26},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":6.7370000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":12.647},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":10.426}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":16.93},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":8.9670000000000005}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":17.699999999999999},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":11.724}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":18.501999999999999},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":13.593999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.385 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.386 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-06T07:57:27.431447+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=27, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.386 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-06T07:57:18.326272+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=7, tm_min=57, tm_sec=18, tm_wday=2, tm_yday=340, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=6, tm_hour=8, tm_min=4, tm_sec=42, tm_wday=2, tm_yday=340, tm_isdst=0) 2023-12-06T08:04:50.386 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-06T08:05:05.149 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:01.581 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.150 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:01.583 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.166 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:01.597 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.193 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:59.147 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.196 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:59.149 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.196 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:46.860 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.197 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:46.861 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.210 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:02:59.164 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:05.211 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:01:46.875 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:10.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-06T08:05:10.952 INFO:teuthology.orchestra.run.smithi084.stdout: 2023-12-06T08:05:10.952 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2023-12-06T08:05:10.964 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2023-12-06T08:05:09.882811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229076,"kb_used_data":7760,"kb_used_omap":15,"kb_used_meta":217008,"kb_avail":754745644,"statfs":{"total":773094113280,"available":772859539456,"internally_reserved":0,"allocated":7946240,"data_stored":3403912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15564,"internal_metadata":222217012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002463"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:52.508005+0000","last_change":"2023-12-06T08:04:52.508666+0000","last_active":"2023-12-06T08:04:52.508005+0000","last_peered":"2023-12-06T08:04:52.508005+0000","last_clean":"2023-12-06T08:04:52.508005+0000","last_became_active":"2023-12-06T08:04:12.344386+0000","last_became_peered":"2023-12-06T08:04:12.344386+0000","last_unstale":"2023-12-06T08:04:52.508005+0000","last_undegraded":"2023-12-06T08:04:52.508005+0000","last_fullsized":"2023-12-06T08:04:52.508005+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:52.506610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:52.506610+0000","last_clean_scrub_stamp":"2023-12-06T08:04:52.506610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T17:43:08.642130+0000","scrub_duration":0.087435324999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:56.470316+0000","last_change":"2023-12-06T08:04:56.471014+0000","last_active":"2023-12-06T08:04:56.470316+0000","last_peered":"2023-12-06T08:04:56.470316+0000","last_clean":"2023-12-06T08:04:56.470316+0000","last_became_active":"2023-12-06T08:04:12.343618+0000","last_became_peered":"2023-12-06T08:04:12.343618+0000","last_unstale":"2023-12-06T08:04:56.470316+0000","last_undegraded":"2023-12-06T08:04:56.470316+0000","last_fullsized":"2023-12-06T08:04:56.470316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:56.468715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:56.468715+0000","last_clean_scrub_stamp":"2023-12-06T08:04:56.468715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:18:21.553988+0000","scrub_duration":0.134725758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:51.445936+0000","last_change":"2023-12-06T08:04:51.446645+0000","last_active":"2023-12-06T08:04:51.445936+0000","last_peered":"2023-12-06T08:04:51.445936+0000","last_clean":"2023-12-06T08:04:51.445936+0000","last_became_active":"2023-12-06T07:58:33.840952+0000","last_became_peered":"2023-12-06T07:58:33.840952+0000","last_unstale":"2023-12-06T08:04:51.445936+0000","last_undegraded":"2023-12-06T08:04:51.445936+0000","last_fullsized":"2023-12-06T08:04:51.445936+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:51.444090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:51.444090+0000","last_clean_scrub_stamp":"2023-12-06T08:04:51.444090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:32:22.481246+0000","scrub_duration":0.030470812999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:49.611884+0000","last_change":"2023-12-06T08:04:49.616293+0000","last_active":"2023-12-06T08:04:49.611884+0000","last_peered":"2023-12-06T08:04:49.611884+0000","last_clean":"2023-12-06T08:04:49.611884+0000","last_became_active":"2023-12-06T08:04:12.481864+0000","last_became_peered":"2023-12-06T08:04:12.481864+0000","last_unstale":"2023-12-06T08:04:49.611884+0000","last_undegraded":"2023-12-06T08:04:49.611884+0000","last_fullsized":"2023-12-06T08:04:49.611884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:49.608094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:49.608094+0000","last_clean_scrub_stamp":"2023-12-06T08:04:49.608094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T11:08:50.343848+0000","scrub_duration":0.30627717100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:49.983545+0000","last_change":"2023-12-06T08:04:49.987962+0000","last_active":"2023-12-06T08:04:49.983545+0000","last_peered":"2023-12-06T08:04:49.983545+0000","last_clean":"2023-12-06T08:04:49.983545+0000","last_became_active":"2023-12-06T08:02:14.378003+0000","last_became_peered":"2023-12-06T08:02:14.378003+0000","last_unstale":"2023-12-06T08:04:49.983545+0000","last_undegraded":"2023-12-06T08:04:49.983545+0000","last_fullsized":"2023-12-06T08:04:49.983545+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:49.978690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:49.978690+0000","last_clean_scrub_stamp":"2023-12-06T08:04:49.978690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T12:29:23.046574+0000","scrub_duration":0.20400399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:50.516417+0000","last_change":"2023-12-06T08:04:50.522607+0000","last_active":"2023-12-06T08:04:50.516417+0000","last_peered":"2023-12-06T08:04:50.516417+0000","last_clean":"2023-12-06T08:04:50.516417+0000","last_became_active":"2023-12-06T08:04:12.538879+0000","last_became_peered":"2023-12-06T08:04:12.538879+0000","last_unstale":"2023-12-06T08:04:50.516417+0000","last_undegraded":"2023-12-06T08:04:50.516417+0000","last_fullsized":"2023-12-06T08:04:50.516417+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:50.512808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:50.512808+0000","last_clean_scrub_stamp":"2023-12-06T08:04:50.512808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T19:33:48.562214+0000","scrub_duration":0.22974007499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":98,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:49.711294+0000","last_change":"2023-12-06T08:04:49.717482+0000","last_active":"2023-12-06T08:04:49.711294+0000","last_peered":"2023-12-06T08:04:49.711294+0000","last_clean":"2023-12-06T08:04:49.711294+0000","last_became_active":"2023-12-06T08:00:11.352381+0000","last_became_peered":"2023-12-06T08:00:11.352381+0000","last_unstale":"2023-12-06T08:04:49.711294+0000","last_undegraded":"2023-12-06T08:04:49.711294+0000","last_fullsized":"2023-12-06T08:04:49.711294+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:49.707660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:49.707660+0000","last_clean_scrub_stamp":"2023-12-06T08:04:49.707660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T16:02:44.337918+0000","scrub_duration":0.224780702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:46.073835+0000","last_change":"2023-12-06T08:04:46.080779+0000","last_active":"2023-12-06T08:04:46.073835+0000","last_peered":"2023-12-06T08:04:46.073835+0000","last_clean":"2023-12-06T08:04:46.073835+0000","last_became_active":"2023-12-06T08:04:16.720764+0000","last_became_peered":"2023-12-06T08:04:16.720764+0000","last_unstale":"2023-12-06T08:04:46.073835+0000","last_undegraded":"2023-12-06T08:04:46.073835+0000","last_fullsized":"2023-12-06T08:04:46.073835+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T08:04:46.069686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T08:04:46.069686+0000","last_clean_scrub_stamp":"2023-12-06T08:04:46.069686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T13:48:25.146612+0000","scrub_duration":0.39180527300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"127'78","reported_seq":246,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-06T08:04:49.094115+0000","last_change":"2023-12-06T08:04:49.101643+0000","last_active":"2023-12-06T08:04:49.094115+0000","last_peered":"2023-12-06T08:04:49.094115+0000","last_clean":"2023-12-06T08:04:49.094115+0000","last_became_active":"2023-12-06T08:04:12.551758+0000","last_became_peered":"2023-12-06T08:04:12.551758+0000","last_unstale":"2023-12-06T08:04:49.094115+0000","last_undegraded":"2023-12-06T08:04:49.094115+0000","last_fullsized":"2023-12-06T08:04:49.094115+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'78","last_scrub_stamp":"2023-12-06T08:04:49.090229+0000","last_deep_scrub":"127'78","last_deep_scrub_stamp":"2023-12-06T08:04:49.090229+0000","last_clean_scrub_stamp":"2023-12-06T08:04:49.090229+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T18:34:35.125909+0000","scrub_duration":0.66232947900000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":11.225,"5min":9.0359999999999996,"15min":9.4039999999999999},"min":{"1min":5.0410000000000004,"5min":3.7370000000000001,"15min":3.7370000000000001},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":29.463999999999999},"last":7.3470000000000004},{"interface":"front","average":{"1min":8.8819999999999997,"5min":9.6549999999999994,"15min":9.0340000000000007},"min":{"1min":5.1520000000000001,"5min":4.2439999999999998,"15min":4.2439999999999998},"max":{"1min":18.582999999999998,"5min":26.544,"15min":26.544},"last":4.4400000000000004}]},{"osd":1,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.262,"5min":10.811,"15min":12.153},"min":{"1min":5.6929999999999996,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":16.93,"5min":30.872,"15min":30.872},"last":10.938000000000001},{"interface":"front","average":{"1min":12.356999999999999,"5min":12.372999999999999,"15min":13.851000000000001},"min":{"1min":7.5899999999999999,"5min":6.375,"15min":6.375},"max":{"1min":21.405999999999999,"5min":30.204000000000001,"15min":39.337000000000003},"last":11.68}]},{"osd":2,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":10.859999999999999,"5min":13.055999999999999,"15min":12.672000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0629999999999997},"max":{"1min":22.399000000000001,"5min":30.273,"15min":30.273},"last":8.8200000000000003},{"interface":"front","average":{"1min":12.191000000000001,"5min":13.754,"15min":14.050000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":6.7569999999999997},"max":{"1min":24.422999999999998,"5min":71.997,"15min":71.997},"last":13.988}]},{"osd":4,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":14.371,"5min":14.709,"15min":13.821},"min":{"1min":7.9720000000000004,"5min":6.5960000000000001,"15min":6.5960000000000001},"max":{"1min":24.727,"5min":31.349,"15min":31.349},"last":8.282},{"interface":"front","average":{"1min":14.209,"5min":15.007,"15min":15.756},"min":{"1min":8.5950000000000006,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":25.234000000000002,"5min":37.817999999999998,"15min":54.225999999999999},"last":15.337}]},{"osd":5,"last update":"Wed Dec 6 08:02:31 2023","interfaces":[{"interface":"back","average":{"1min":16.190000000000001,"5min":16.385000000000002,"15min":16.943999999999999},"min":{"1min":7.6799999999999997,"5min":7.6799999999999997,"15min":7.6799999999999997},"max":{"1min":27.396999999999998,"5min":50.066000000000003,"15min":50.066000000000003},"last":12.253},{"interface":"front","average":{"1min":16.195,"5min":17.016999999999999,"15min":17.27},"min":{"1min":12.411,"5min":8.3870000000000005,"15min":8.3870000000000005},"max":{"1min":24.518000000000001,"5min":50.478000000000002,"15min":50.478000000000002},"last":13.215}]},{"osd":6,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":15.304,"15min":14.879},"min":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.3680000000000003},"max":{"1min":23.588999999999999,"5min":31.768000000000001,"15min":31.768000000000001},"last":14.726000000000001},{"interface":"front","average":{"1min":15.83,"5min":17.797000000000001,"15min":18.513000000000002},"min":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.4299999999999997},"max":{"1min":28.294,"5min":37.182000000000002,"15min":44.255000000000003},"last":15.879}]},{"osd":7,"last update":"Wed Dec 6 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":17.187000000000001,"5min":16.628,"15min":18.411999999999999},"min":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"max":{"1min":31.565000000000001,"5min":54.515999999999998,"15min":54.515999999999998},"last":10.542},{"interface":"front","average":{"1min":14.093,"5min":17.045000000000002,"15min":18.084},"min":{"1min":9.3480000000000008,"5min":9.3480000000000008,"15min":9.2330000000000005},"max":{"1min":26.427,"5min":51.012999999999998,"15min":51.012999999999998},"last":16.997}]}]},{"osd":2,"up_from":16,"seq":68719476834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":9.8439999999999994,"15min":10.177},"min":{"1min":4.2050000000000001,"5min":4.2050000000000001,"15min":4.2050000000000001},"max":{"1min":22.251999999999999,"5min":40.771999999999998,"15min":40.771999999999998},"last":10.369999999999999},{"interface":"front","average":{"1min":8.4399999999999995,"5min":9.3480000000000008,"15min":10.211},"min":{"1min":5.673,"5min":4.4779999999999998,"15min":4.4779999999999998},"max":{"1min":14.021000000000001,"5min":45.781999999999996,"15min":45.781999999999996},"last":5.5579999999999998}]},{"osd":1,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.792999999999999,"5min":12.814,"15min":18.655999999999999},"min":{"1min":6.8959999999999999,"5min":5.657,"15min":5.657},"max":{"1min":39.911999999999999,"5min":39.911999999999999,"15min":165.68600000000001},"last":12.167999999999999},{"interface":"front","average":{"1min":13.313000000000001,"5min":13.093,"15min":19.096},"min":{"1min":5.923,"5min":5.5659999999999998,"15min":5.5659999999999998},"max":{"1min":30.786999999999999,"5min":30.786999999999999,"15min":153.68199999999999},"last":7.0860000000000003}]},{"osd":3,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.664999999999999,"5min":11.467000000000001,"15min":11.18},"min":{"1min":7.0490000000000004,"5min":5.2629999999999999,"15min":5.2629999999999999},"max":{"1min":40.226999999999997,"5min":40.226999999999997,"15min":40.226999999999997},"last":10.923999999999999},{"interface":"front","average":{"1min":16.241,"5min":15.218,"15min":15.601000000000001},"min":{"1min":9.0559999999999992,"5min":7.3040000000000003,"15min":7.3040000000000003},"max":{"1min":36.771999999999998,"5min":36.771999999999998,"15min":36.771999999999998},"last":11.568}]},{"osd":4,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":13.721,"5min":13.516,"15min":15.509},"min":{"1min":8.048,"5min":6.0179999999999998,"15min":6.0179999999999998},"max":{"1min":23.898,"5min":48.207999999999998,"15min":50.137999999999998},"last":14.220000000000001},{"interface":"front","average":{"1min":17.294,"5min":17.658000000000001,"15min":16.991},"min":{"1min":9.8680000000000003,"5min":9.5510000000000002,"15min":7.1429999999999998},"max":{"1min":29.786000000000001,"5min":51.164999999999999,"15min":51.164999999999999},"last":13.302}]},{"osd":5,"last update":"Wed Dec 6 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":20.486999999999998,"5min":19.033999999999999,"15min":20.097999999999999},"min":{"1min":12.346,"5min":9.3339999999999996,"15min":9.3339999999999996},"max":{"1min":34.911999999999999,"5min":50.320999999999998,"15min":50.320999999999998},"last":14.893000000000001},{"interface":"front","average":{"1min":15.222,"5min":15.279999999999999,"15min":16.949999999999999},"min":{"1min":9.0969999999999995,"5min":5.9619999999999997,"15min":5.9619999999999997},"max":{"1min":26.042999999999999,"5min":44.323,"15min":44.323},"last":15.585000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":17.036000000000001,"5min":15.763999999999999,"15min":18.670999999999999},"min":{"1min":8.8010000000000002,"5min":7.71,"15min":7.71},"max":{"1min":31.346,"5min":44.640999999999998,"15min":104.834},"last":16.222000000000001},{"interface":"front","average":{"1min":20.312999999999999,"5min":19.507999999999999,"15min":20.448},"min":{"1min":12.666,"5min":11.135,"15min":11.135},"max":{"1min":33.591000000000001,"5min":43.287999999999997,"15min":49.146999999999998},"last":16.826000000000001}]},{"osd":7,"last update":"Wed Dec 6 08:04:43 2023","interfaces":[{"interface":"back","average":{"1min":16.402000000000001,"5min":16.791,"15min":18.712},"min":{"1min":8.3320000000000007,"5min":7.0170000000000003,"15min":7.0170000000000003},"max":{"1min":35.655000000000001,"5min":35.655000000000001,"15min":42.915999999999997},"last":8.4019999999999992},{"interface":"front","average":{"1min":18.847999999999999,"5min":17.591000000000001,"15min":18.648},"min":{"1min":10.771000000000001,"5min":9.2289999999999992,"15min":9.2289999999999992},"max":{"1min":35.119999999999997,"5min":43.697000000000003,"15min":43.697000000000003},"last":17.419}]}]},{"osd":1,"up_from":16,"seq":68719476834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":14.696999999999999,"5min":11.881,"15min":12.816000000000001},"min":{"1min":5.2190000000000003,"5min":4.835,"15min":4.835},"max":{"1min":86.849999999999994,"5min":86.849999999999994,"15min":86.849999999999994},"last":11.989000000000001},{"interface":"front","average":{"1min":16.329999999999998,"5min":14.273,"15min":12.353},"min":{"1min":8.5570000000000004,"5min":4.7999999999999998,"15min":4.7999999999999998},"max":{"1min":76.355000000000004,"5min":76.355000000000004,"15min":76.355000000000004},"last":14.276}]},{"osd":2,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":14.247,"15min":18.376999999999999},"min":{"1min":8.7599999999999998,"5min":7.0940000000000003,"15min":6.1280000000000001},"max":{"1min":24.52,"5min":32.081000000000003,"15min":100.703},"last":12.772},{"interface":"front","average":{"1min":8.0730000000000004,"5min":9.3610000000000007,"15min":15.763},"min":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":17.173999999999999,"5min":29.946999999999999,"15min":95.960999999999999},"last":5.4470000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":16.475999999999999,"15min":17.669},"min":{"1min":9.9329999999999998,"5min":9.9329999999999998,"15min":8.0250000000000004},"max":{"1min":26.606999999999999,"5min":44.055,"15min":44.055},"last":13.547000000000001},{"interface":"front","average":{"1min":8.3390000000000004,"5min":9.9789999999999992,"15min":12.343},"min":{"1min":5.5449999999999999,"5min":5.5449999999999999,"15min":5.5449999999999999},"max":{"1min":23.137,"5min":42.786000000000001,"15min":46.613},"last":7.9260000000000002}]},{"osd":4,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":15.102,"5min":15.279999999999999,"15min":17.399000000000001},"min":{"1min":10.817,"5min":6.6020000000000003,"15min":6.6020000000000003},"max":{"1min":27.620999999999999,"5min":32.173000000000002,"15min":81.838999999999999},"last":15.055},{"interface":"front","average":{"1min":17.588000000000001,"5min":18.548999999999999,"15min":20.359999999999999},"min":{"1min":13.970000000000001,"5min":10.754,"15min":10.313000000000001},"max":{"1min":28.974,"5min":33.311,"15min":66.515000000000001},"last":16.707999999999998}]},{"osd":5,"last update":"Wed Dec 6 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":17.401,"5min":14.557,"15min":15.941000000000001},"min":{"1min":6.4619999999999997,"5min":6.351,"15min":6.351},"max":{"1min":56.423000000000002,"5min":56.423000000000002,"15min":56.423000000000002},"last":6.6369999999999996},{"interface":"front","average":{"1min":13.257,"5min":15.893000000000001,"15min":18.628},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":54.606000000000002,"5min":82.381,"15min":82.381},"last":15.81}]},{"osd":6,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":11.569000000000001,"5min":14.236000000000001,"15min":17},"min":{"1min":6.4820000000000002,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.640000000000001,"5min":31.954999999999998,"15min":58.619},"last":9.6349999999999998},{"interface":"front","average":{"1min":10.394,"5min":13.231999999999999,"15min":18.577000000000002},"min":{"1min":5.8780000000000001,"5min":5.8769999999999998,"15min":5.8769999999999998},"max":{"1min":21.196000000000002,"5min":33.671999999999997,"15min":83.632000000000005},"last":8.8320000000000007}]},{"osd":7,"last update":"Wed Dec 6 08:04:35 2023","interfaces":[{"interface":"back","average":{"1min":18.204999999999998,"5min":19.140999999999998,"15min":21.466999999999999},"min":{"1min":12.698,"5min":12.069000000000001,"15min":12.069000000000001},"max":{"1min":32.790999999999997,"5min":34.576999999999998,"15min":62.006999999999998},"last":17.472000000000001},{"interface":"front","average":{"1min":11.766,"5min":14.564,"15min":18.484999999999999},"min":{"1min":7.4500000000000002,"5min":7.4500000000000002,"15min":7.4500000000000002},"max":{"1min":32.295000000000002,"5min":32.512999999999998,"15min":60.698999999999998},"last":10.523}]}]},{"osd":0,"up_from":16,"seq":68719476836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":7.9240000000000004,"5min":10.243,"15min":16.459},"min":{"1min":4.6879999999999997,"5min":3.7229999999999999,"15min":3.7229999999999999},"max":{"1min":12.43,"5min":64.242999999999995,"15min":252.88399999999999},"last":14.023},{"interface":"front","average":{"1min":9.8780000000000001,"5min":11.519,"15min":17.565000000000001},"min":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"max":{"1min":19.867000000000001,"5min":63.192999999999998,"15min":274.91500000000002},"last":16.388999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":12.132,"5min":12.692,"15min":15.661},"min":{"1min":7.5030000000000001,"5min":5.093,"15min":5.093},"max":{"1min":24.282,"5min":35.383000000000003,"15min":82.866},"last":9.7159999999999993},{"interface":"front","average":{"1min":13.413,"5min":13.98,"15min":15.507},"min":{"1min":7.8280000000000003,"5min":6.6980000000000004,"15min":6.6980000000000004},"max":{"1min":23.916,"5min":34.289999999999999,"15min":85.991},"last":6.7690000000000001}]},{"osd":3,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":8.8049999999999997,"5min":10.826000000000001,"15min":13.613},"min":{"1min":5.984,"5min":5.984,"15min":4.5970000000000004},"max":{"1min":14.737,"5min":41.045000000000002,"15min":76.593000000000004},"last":7.6879999999999997},{"interface":"front","average":{"1min":13.013999999999999,"5min":15.478,"15min":18.724},"min":{"1min":8.4619999999999997,"5min":7.5890000000000004,"15min":6.9960000000000004},"max":{"1min":16.349,"5min":42.284999999999997,"15min":85.162000000000006},"last":11.461}]},{"osd":4,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":14.909000000000001,"5min":15.869999999999999,"15min":16.788},"min":{"1min":8.9260000000000002,"5min":5.9729999999999999,"15min":5.9729999999999999},"max":{"1min":26.155000000000001,"5min":76.387,"15min":76.387},"last":12.164},{"interface":"front","average":{"1min":11.792999999999999,"5min":13.420999999999999,"15min":13.662000000000001},"min":{"1min":6.9669999999999996,"5min":5.8390000000000004,"15min":5.8390000000000004},"max":{"1min":18.969000000000001,"5min":29.375,"15min":70.085999999999999},"last":10.475}]},{"osd":5,"last update":"Wed Dec 6 08:02:33 2023","interfaces":[{"interface":"back","average":{"1min":18.073,"5min":19.533999999999999,"15min":25.425999999999998},"min":{"1min":9.7119999999999997,"5min":8.2119999999999997,"15min":8.2119999999999997},"max":{"1min":35.901000000000003,"5min":144.60499999999999,"15min":144.60499999999999},"last":17.088000000000001},{"interface":"front","average":{"1min":20.181999999999999,"5min":19.937999999999999,"15min":24.448},"min":{"1min":10.061,"5min":8.4610000000000003,"15min":8.4610000000000003},"max":{"1min":36.506,"5min":142.25,"15min":142.25},"last":8.2729999999999997}]},{"osd":6,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":14.260999999999999,"5min":16.559999999999999,"15min":17.452999999999999},"min":{"1min":9.7309999999999999,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":19.756,"5min":31.550000000000001,"15min":75.471999999999994},"last":14.916},{"interface":"front","average":{"1min":16.353000000000002,"5min":17.459,"15min":21.631},"min":{"1min":10.074999999999999,"5min":7.8760000000000003,"15min":7.8760000000000003},"max":{"1min":27.234999999999999,"5min":37.479999999999997,"15min":84.617000000000004},"last":13.353}]},{"osd":7,"last update":"Wed Dec 6 08:04:37 2023","interfaces":[{"interface":"back","average":{"1min":15.933,"5min":19.382000000000001,"15min":20.969999999999999},"min":{"1min":8.9049999999999994,"5min":8.9049999999999994,"15min":7.6219999999999999},"max":{"1min":28.501000000000001,"5min":78.506,"15min":84.031000000000006},"last":15.821},{"interface":"front","average":{"1min":16.109000000000002,"5min":17.524000000000001,"15min":18.382999999999999},"min":{"1min":9.7370000000000001,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":21.628,"5min":49.362000000000002,"15min":77.974000000000004},"last":18.207000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.0220000000000002,"5min":8.984,"15min":9.3019999999999996},"min":{"1min":3.8500000000000001,"5min":3.8500000000000001,"15min":3.8500000000000001},"max":{"1min":15.817,"5min":48.454000000000001,"15min":48.454000000000001},"last":18.827000000000002},{"interface":"front","average":{"1min":9.9770000000000003,"5min":9.907,"15min":10.127000000000001},"min":{"1min":4.5119999999999996,"5min":4.4390000000000001,"15min":4.4390000000000001},"max":{"1min":18.184999999999999,"5min":44.947000000000003,"15min":44.947000000000003},"last":39.061}]},{"osd":1,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":9.3599999999999994,"5min":10.111000000000001,"15min":11.243},"min":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.0090000000000003},"max":{"1min":17.593,"5min":34.880000000000003,"15min":35.524000000000001},"last":12.657},{"interface":"front","average":{"1min":11.289,"5min":11.656000000000001,"15min":13.186},"min":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":5.6269999999999998},"max":{"1min":20.495000000000001,"5min":26.003,"15min":31.027000000000001},"last":9.843}]},{"osd":2,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":11.337,"5min":13.117000000000001,"15min":12.476000000000001},"min":{"1min":5.6440000000000001,"5min":5.6440000000000001,"15min":5.6440000000000001},"max":{"1min":19.222999999999999,"5min":29.806999999999999,"15min":29.806999999999999},"last":14.887},{"interface":"front","average":{"1min":13.704000000000001,"5min":14.004,"15min":14.481},"min":{"1min":7.2770000000000001,"5min":6.8929999999999998,"15min":6.8929999999999998},"max":{"1min":23.795999999999999,"5min":27.408000000000001,"15min":32.915999999999997},"last":14.15}]},{"osd":3,"last update":"Wed Dec 6 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":12.711,"5min":12.853999999999999,"15min":11.895},"min":{"1min":6.4379999999999997,"5min":6.1180000000000003,"15min":6.1180000000000003},"max":{"1min":25.167000000000002,"5min":28.991,"15min":28.991},"last":7.8019999999999996},{"interface":"front","average":{"1min":14.364000000000001,"5min":15.183,"15min":14.550000000000001},"min":{"1min":9.4640000000000004,"5min":7.3730000000000002,"15min":7.3730000000000002},"max":{"1min":20.163,"5min":35.540999999999997,"15min":35.540999999999997},"last":16.423999999999999}]},{"osd":5,"last update":"Wed Dec 6 08:02:25 2023","interfaces":[{"interface":"back","average":{"1min":18.305,"5min":17.25,"15min":17.149000000000001},"min":{"1min":9.7899999999999991,"5min":7.5940000000000003,"15min":7.5940000000000003},"max":{"1min":32.302999999999997,"5min":53.899999999999999,"15min":53.899999999999999},"last":17.771000000000001},{"interface":"front","average":{"1min":19.698,"5min":17.777999999999999,"15min":17.704999999999998},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.9740000000000002},"max":{"1min":34.222000000000001,"5min":55.914000000000001,"15min":55.914000000000001},"last":15.602}]},{"osd":6,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":14.391999999999999,"5min":15.055,"15min":19.649999999999999},"min":{"1min":7.7690000000000001,"5min":6.3769999999999998,"15min":6.3769999999999998},"max":{"1min":25.029,"5min":45.374000000000002,"15min":93.073999999999998},"last":12.143000000000001},{"interface":"front","average":{"1min":16.972999999999999,"5min":16.369,"15min":17.535},"min":{"1min":11.090999999999999,"5min":6.7779999999999996,"15min":6.7779999999999996},"max":{"1min":25.626000000000001,"5min":43.737000000000002,"15min":91.075999999999993},"last":11.497}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.245000000000001,"5min":17.48,"15min":16.675999999999998},"min":{"1min":8.4260000000000002,"5min":8.4260000000000002,"15min":8.4260000000000002},"max":{"1min":19.492999999999999,"5min":75.102000000000004,"15min":75.102000000000004},"last":19.582000000000001},{"interface":"front","average":{"1min":16.640999999999998,"5min":18.725000000000001,"15min":17.587},"min":{"1min":10.114000000000001,"5min":10.114000000000001,"15min":10.114000000000001},"max":{"1min":24.939,"5min":110.646,"15min":110.646},"last":10.839}]}]},{"osd":6,"up_from":14,"seq":60129542242,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":7.3769999999999998,"5min":9.0990000000000002,"15min":9.8109999999999999},"min":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"max":{"1min":17.864000000000001,"5min":35.454000000000001,"15min":70.965000000000003},"last":7.5919999999999996},{"interface":"front","average":{"1min":9.5459999999999994,"5min":10.318,"15min":11.42},"min":{"1min":5.8019999999999996,"5min":4.2279999999999998,"15min":4.2279999999999998},"max":{"1min":17.193999999999999,"5min":56.200000000000003,"15min":73.582999999999998},"last":6.1520000000000001}]},{"osd":1,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":13.073,"15min":16.977},"min":{"1min":5.4340000000000002,"5min":4.9900000000000002,"15min":4.383},"max":{"1min":48.749000000000002,"5min":70.049000000000007,"15min":217.595},"last":8.343},{"interface":"front","average":{"1min":12.454000000000001,"5min":12.571,"15min":19.001000000000001},"min":{"1min":6.7779999999999996,"5min":5.2930000000000001,"15min":5.2930000000000001},"max":{"1min":50.026000000000003,"5min":50.026000000000003,"15min":284.96499999999997},"last":17.315999999999999}]},{"osd":2,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.516999999999999,"5min":13.451000000000001,"15min":13.664},"min":{"1min":6.7590000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.384,"5min":61.747,"15min":61.747},"last":8.9649999999999999},{"interface":"front","average":{"1min":13.880000000000001,"5min":14.494999999999999,"15min":15.858000000000001},"min":{"1min":6.9240000000000004,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":24.280999999999999,"5min":55.357999999999997,"15min":55.357999999999997},"last":11.699999999999999}]},{"osd":3,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":12.087,"5min":16.050000000000001,"15min":15.833},"min":{"1min":7.7869999999999999,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":21.626000000000001,"5min":73.149000000000001,"15min":73.149000000000001},"last":9.5800000000000001},{"interface":"front","average":{"1min":16.533000000000001,"5min":17.658000000000001,"15min":21.414999999999999},"min":{"1min":9.0630000000000006,"5min":6.0979999999999999,"15min":6.0979999999999999},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":189.84999999999999},"last":10.364000000000001}]},{"osd":4,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":16.527999999999999,"5min":18.452999999999999,"15min":16.699999999999999},"min":{"1min":7.4459999999999997,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":64.989000000000004,"5min":66.349999999999994,"15min":66.349999999999994},"last":11.090999999999999},{"interface":"front","average":{"1min":13.773999999999999,"5min":16.446999999999999,"15min":15.271000000000001},"min":{"1min":9.1539999999999999,"5min":5.8710000000000004,"15min":5.8710000000000004},"max":{"1min":21.715,"5min":70.138999999999996,"15min":70.138999999999996},"last":12.333}]},{"osd":5,"last update":"Wed Dec 6 08:02:23 2023","interfaces":[{"interface":"back","average":{"1min":22.119,"5min":19.501999999999999,"15min":19.116},"min":{"1min":9.3800000000000008,"5min":9.3800000000000008,"15min":9.3800000000000008},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":18.108000000000001},{"interface":"front","average":{"1min":14.542999999999999,"5min":14.630000000000001,"15min":14.458},"min":{"1min":5.6079999999999997,"5min":5.6079999999999997,"15min":5.6079999999999997},"max":{"1min":56.968000000000004,"5min":56.968000000000004,"15min":56.968000000000004},"last":15.492000000000001}]},{"osd":7,"last update":"Wed Dec 6 08:04:29 2023","interfaces":[{"interface":"back","average":{"1min":17.797999999999998,"5min":18.384,"15min":16.677},"min":{"1min":10.664999999999999,"5min":8.4939999999999998,"15min":7.6740000000000004},"max":{"1min":65.936999999999998,"5min":70.813000000000002,"15min":70.813000000000002},"last":16.131},{"interface":"front","average":{"1min":17.844000000000001,"5min":20.321000000000002,"15min":19.661999999999999},"min":{"1min":10.233000000000001,"5min":7.7009999999999996,"15min":7.7009999999999996},"max":{"1min":75.370000000000005,"5min":75.370000000000005,"15min":75.370000000000005},"last":14.505000000000001}]}]},{"osd":5,"up_from":108,"seq":463856467982,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":1264,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":1294336,"data_stored":723869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28634824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.051}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6579999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.701000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.81}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.087}]}]},{"osd":7,"up_from":13,"seq":55834574945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":326029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":7.5519999999999996,"5min":8.1590000000000007,"15min":9.0709999999999997},"min":{"1min":5.2460000000000004,"5min":4.7590000000000003,"15min":4.7590000000000003},"max":{"1min":10.032999999999999,"5min":26.225000000000001,"15min":26.225000000000001},"last":7.3289999999999997},{"interface":"front","average":{"1min":11.515000000000001,"5min":12.269,"15min":12.125999999999999},"min":{"1min":7.1710000000000003,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":28.321999999999999,"5min":28.321999999999999,"15min":28.321999999999999},"last":12.602}]},{"osd":1,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":12.493,"5min":13.238,"15min":15.345000000000001},"min":{"1min":6.2910000000000004,"5min":6.2910000000000004,"15min":6.0890000000000004},"max":{"1min":23.324999999999999,"5min":38.838999999999999,"15min":51.890000000000001},"last":10.667999999999999},{"interface":"front","average":{"1min":7.7800000000000002,"5min":9.0239999999999991,"15min":8.6150000000000002},"min":{"1min":4.968,"5min":4.1260000000000003,"15min":4.1260000000000003},"max":{"1min":12.625,"5min":37.042999999999999,"15min":37.042999999999999},"last":9.8770000000000007}]},{"osd":2,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":9.6120000000000001,"5min":11.505000000000001,"15min":12.268000000000001},"min":{"1min":5.6100000000000003,"5min":4.359,"15min":4.359},"max":{"1min":15.218,"5min":30.379999999999999,"15min":35.225999999999999},"last":18.693000000000001},{"interface":"front","average":{"1min":11.488,"5min":12.32,"15min":11.657999999999999},"min":{"1min":6.4390000000000001,"5min":6.3650000000000002,"15min":5.7590000000000003},"max":{"1min":16.571000000000002,"5min":35.002000000000002,"15min":35.002000000000002},"last":16.581}]},{"osd":3,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":10.773,"5min":12.019,"15min":23.132000000000001},"min":{"1min":6.1399999999999997,"5min":6.1399999999999997,"15min":6.1399999999999997},"max":{"1min":17.382999999999999,"5min":27.995999999999999,"15min":228.17099999999999},"last":15.978},{"interface":"front","average":{"1min":13.83,"5min":15.121,"15min":25.960000000000001},"min":{"1min":7.3319999999999999,"5min":7.3319999999999999,"15min":7.3319999999999999},"max":{"1min":24.050999999999998,"5min":35.183999999999997,"15min":270.13},"last":17.469999999999999}]},{"osd":4,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":14.162000000000001,"15min":16.097999999999999},"min":{"1min":6.7469999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"max":{"1min":33.759999999999998,"5min":40.521000000000001,"15min":86.980999999999995},"last":20.460000000000001},{"interface":"front","average":{"1min":13.933,"5min":15.076000000000001,"15min":18.712},"min":{"1min":8.2560000000000002,"5min":6.9660000000000002,"15min":6.9660000000000002},"max":{"1min":36.215000000000003,"5min":36.215000000000003,"15min":99.021000000000001},"last":8.9109999999999996}]},{"osd":5,"last update":"Wed Dec 6 08:02:29 2023","interfaces":[{"interface":"back","average":{"1min":19.053000000000001,"5min":18.925000000000001,"15min":21.117000000000001},"min":{"1min":11.496,"5min":10.254,"15min":10.254},"max":{"1min":41.997999999999998,"5min":97.191000000000003,"15min":97.191000000000003},"last":18.719999999999999},{"interface":"front","average":{"1min":16.82,"5min":17.844999999999999,"15min":20.274999999999999},"min":{"1min":9.7840000000000007,"5min":8.6609999999999996,"15min":8.6609999999999996},"max":{"1min":33.161999999999999,"5min":64.048000000000002,"15min":64.048000000000002},"last":12.755000000000001}]},{"osd":6,"last update":"Wed Dec 6 08:04:31 2023","interfaces":[{"interface":"back","average":{"1min":14.776999999999999,"5min":15.06,"15min":16.565000000000001},"min":{"1min":9.8569999999999993,"5min":6.2469999999999999,"15min":6.2469999999999999},"max":{"1min":21.498000000000001,"5min":36.119999999999997,"15min":74.064999999999998},"last":21.286000000000001},{"interface":"front","average":{"1min":15.085000000000001,"5min":17.966000000000001,"15min":20.872},"min":{"1min":10.289,"5min":10.055,"15min":10.055},"max":{"1min":20.605,"5min":42.814,"15min":92.477999999999994},"last":18.163}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T08:05:10.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-06T08:05:11.667 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-06T08:05:11.667 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-06T08:05:11.668 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-06T08:05:11.668 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:05:11.676 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:05:11.668+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86974) UID: 0 2023-12-06T08:05:11.676 INFO:tasks.ceph.osd.0.smithi084.stderr:2023-12-06T08:05:11.669+0000 b47e640 -1 osd.0 127 *** Got signal Terminated *** 2023-12-06T08:05:25.185 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:21.617 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.188 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:21.621 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.204 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:21.636 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.238 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:19.192 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.241 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:19.195 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.252 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:06.916 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.255 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:06.918 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.256 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:19.210 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:25.270 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:06.933 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.230 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:41.663 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.233 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:41.666 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.246 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:04:41.679 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.283 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:39.237 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.284 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:26.948 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.285 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:39.239 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.286 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:26.950 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.300 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:39.254 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:45.301 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:26.965 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:05:58.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-06T08:06:00.481 INFO:tasks.ceph.osd.0:Stopped 2023-12-06T08:06:00.481 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-06T08:06:00.481 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:06:00.490 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:06:00.482+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86979) UID: 0 2023-12-06T08:06:00.490 INFO:tasks.ceph.osd.1.smithi084.stderr:2023-12-06T08:06:00.482+0000 b47e640 -1 osd.1 129 *** Got signal Terminated *** 2023-12-06T08:06:04.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-06T08:06:05.352 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:01.785 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.354 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:01.787 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.371 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:01.803 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.398 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:59.352 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.399 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:59.353 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.402 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:47.066 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.404 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:47.068 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.414 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:03:59.368 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:05.419 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:02:47.083 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:24.887 INFO:tasks.ceph.osd.1:Stopped 2023-12-06T08:06:24.887 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-06T08:06:24.887 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:06:24.896 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:06:24.888+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86984) UID: 0 2023-12-06T08:06:24.896 INFO:tasks.ceph.osd.2.smithi084.stderr:2023-12-06T08:06:24.889+0000 b47e640 -1 osd.2 133 *** Got signal Terminated *** 2023-12-06T08:06:25.400 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:21.833 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.404 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:21.836 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.418 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:21.850 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.446 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:19.399 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.447 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:19.401 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.451 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:07.114 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.453 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:07.116 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.461 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:19.414 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:25.471 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:07.135 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.552 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:22.984 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.557 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:22.989 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.576 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:23.008 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.607 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:20.561 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.608 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:20.562 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.614 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:08.278 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.615 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:08.279 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.621 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:20.575 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:26.633 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:08.296 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:35.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-06T08:06:37.091 INFO:tasks.ceph.osd.2:Stopped 2023-12-06T08:06:37.091 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-06T08:06:37.091 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:06:37.100 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:06:37.092+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86992) UID: 0 2023-12-06T08:06:37.100 INFO:tasks.ceph.osd.3.smithi084.stderr:2023-12-06T08:06:37.092+0000 b47e640 -1 osd.3 135 *** Got signal Terminated *** 2023-12-06T08:06:41.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-06T08:06:45.397 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:41.829 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.399 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:41.831 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.413 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:05:41.845 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.448 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:27.112 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.449 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:39.402 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.449 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:27.113 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.450 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:39.404 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.464 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:27.127 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:45.465 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:39.418 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:06:49.294 INFO:tasks.ceph.osd.3:Stopped 2023-12-06T08:06:49.295 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-06T08:06:49.295 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:06:49.303 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:06:49.295+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86702) UID: 0 2023-12-06T08:06:49.307 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-12-06T08:06:49.296+0000 b47e640 -1 osd.4 139 *** Got signal Terminated *** 2023-12-06T08:06:59.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-06T08:07:01.498 INFO:tasks.ceph.osd.4:Stopped 2023-12-06T08:07:01.499 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-06T08:07:01.499 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:07:01.512 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:07:01.503+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 113646) UID: 0 2023-12-06T08:07:01.514 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-12-06T08:07:01.505+0000 b47e640 -1 osd.5 141 *** Got signal Terminated *** 2023-12-06T08:07:05.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-06T08:07:05.569 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:02.001 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.572 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:02.005 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.589 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:02.021 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.617 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:59.571 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.618 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:59.572 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.619 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:47.282 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.620 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:47.284 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.636 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:04:59.589 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:05.637 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:03:47.300 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.508 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:21.940 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.513 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:21.945 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.531 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:21.963 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.558 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:19.512 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.563 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:07.227 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.564 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:19.518 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.567 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:07.230 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.578 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:19.532 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.581 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:07.244 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:25.905 INFO:tasks.ceph.osd.5:Stopped 2023-12-06T08:07:25.905 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-06T08:07:25.905 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:07:25.913 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:07:25.905+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86706) UID: 0 2023-12-06T08:07:25.916 INFO:tasks.ceph.osd.6.smithi107.stderr:2023-12-06T08:07:25.906+0000 b47e640 -1 osd.6 145 *** Got signal Terminated *** 2023-12-06T08:07:45.553 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:41.985 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.557 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:41.989 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.573 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:06:42.005 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.605 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:27.268 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.609 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:27.272 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.618 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:39.571 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.622 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:27.286 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.624 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:39.578 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:07:45.641 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:39.595 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.599 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:02.031 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.603 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:02.035 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.617 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:02.050 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.647 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:47.310 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.647 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:59.600 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.650 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:59.603 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.650 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:47.314 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.665 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:04:47.328 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:05.665 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:05:59.619 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:14.718 INFO:tasks.ceph.osd.6:Stopped 2023-12-06T08:08:14.719 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-06T08:08:14.719 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:14.728 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:08:14.720+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86708) UID: 0 2023-12-06T08:08:14.728 INFO:tasks.ceph.osd.7.smithi107.stderr:2023-12-06T08:08:14.721+0000 b47e640 -1 osd.7 145 *** Got signal Terminated *** 2023-12-06T08:08:25.654 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:22.086 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.656 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:22.088 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.671 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:22.104 110507== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.701 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:06:19.654 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.703 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:06:19.657 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.704 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:07.367 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.707 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:07.370 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.716 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:06:19.670 105616== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:25.720 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:07.384 126528== Warning: unimplemented fcntl command: 1036 2023-12-06T08:08:45.227 INFO:tasks.ceph.osd.7:Stopped 2023-12-06T08:08:45.228 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-06T08:08:45.228 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-06T08:08:45.228 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:45.253 INFO:tasks.ceph.mgr.y:Stopped 2023-12-06T08:08:45.253 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-06T08:08:45.253 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:45.255 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-12-06T08:08:45.253+0000 7f08a550d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86161) UID: 0 2023-12-06T08:08:45.281 INFO:tasks.ceph.mgr.x:Stopped 2023-12-06T08:08:45.281 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-06T08:08:45.281 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-06T08:08:45.281 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:45.306 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:08:45.296+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 110502) UID: 0 2023-12-06T08:08:45.307 INFO:tasks.ceph.mon.a.smithi084.stderr:2023-12-06T08:08:45.297+0000 9f54640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T08:08:45.916 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:42.348 110507== 2023-12-06T08:08:45.916 INFO:tasks.ceph.mon.a.smithi084.stderr:==00:00:07:42.348 110507== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:08:45.928 INFO:tasks.ceph.mon.a.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:08:47.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-06T08:08:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:08:51.384 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-06T08:08:51.384 INFO:tasks.ceph.mon.a:Stopped 2023-12-06T08:08:51.385 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-06T08:08:51.385 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:51.400 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:08:51.389+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 126510) UID: 0 2023-12-06T08:08:51.400 INFO:tasks.ceph.mon.c.smithi084.stderr:2023-12-06T08:08:51.390+0000 9f54640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-06T08:08:52.056 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:33.719 126528== 2023-12-06T08:08:52.056 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:05:33.719 126528== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:08:52.068 INFO:tasks.ceph.mon.c.smithi084.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:08:52.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-06T08:08:52.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-06T08:08:57.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2023-12-06T08:08:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:08:57.487 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-06T08:08:57.488 INFO:tasks.ceph.mon.c:Stopped 2023-12-06T08:08:57.488 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-06T08:08:57.489 INFO:teuthology.orchestra.run:waiting for 300 2023-12-06T08:08:57.504 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:08:57.492+0000 9f54640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 105614) UID: 0 2023-12-06T08:08:57.504 INFO:tasks.ceph.mon.b.smithi107.stderr:2023-12-06T08:08:57.494+0000 9f54640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-06T08:08:58.141 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:06:52.094 105616== 2023-12-06T08:08:58.141 INFO:tasks.ceph.mon.b.smithi107.stderr:==00:00:06:52.094 105616== Exit program on first error (--exit-on-first-error=yes) 2023-12-06T08:08:58.152 INFO:tasks.ceph.mon.b.smithi107.stderr:daemon-helper: command failed with exit status 42 2023-12-06T08:09:02.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-06T08:09:02.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-06T08:09:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-06T08:09:03.591 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-06T08:09:03.591 INFO:tasks.ceph.mon.b:Stopped 2023-12-06T08:09:03.591 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-06T08:09:03.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW OPS\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-06T08:09:03.710 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T08:09:03.710 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-06T08:09:03.830 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T08:09:03.830 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-06T08:09:03.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T08:09:03.904 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-06T08:09:03.963 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T08:09:03.963 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-06T08:09:04.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:09:04.020 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-06T08:09:04.103 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:09:04.104 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-06T08:09:04.166 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:09:04.166 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-06T08:09:04.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:09:04.221 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-06T08:09:04.268 INFO:tasks.ceph:Archiving mon data... 2023-12-06T08:09:04.271 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/data/mon.a.tgz 2023-12-06T08:09:04.272 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2023-12-06T08:09:04.287 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.D9Ijt0XRXn 2023-12-06T08:09:04.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.D9Ijt0XRXn 2023-12-06T08:09:04.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.D9Ijt0XRXn 2023-12-06T08:09:04.538 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.D9Ijt0XRXn is 316KB 2023-12-06T08:09:04.635 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.D9Ijt0XRXn 2023-12-06T08:09:04.653 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/data/mon.c.tgz 2023-12-06T08:09:04.653 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2023-12-06T08:09:04.703 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.D6XdxnUS9I 2023-12-06T08:09:04.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.D6XdxnUS9I 2023-12-06T08:09:04.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.D6XdxnUS9I 2023-12-06T08:09:04.909 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.D6XdxnUS9I is 317KB 2023-12-06T08:09:05.004 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.D6XdxnUS9I 2023-12-06T08:09:05.018 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/data/mon.b.tgz 2023-12-06T08:09:05.018 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-12-06T08:09:05.036 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.IEes7jv88b 2023-12-06T08:09:05.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.IEes7jv88b 2023-12-06T08:09:05.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.IEes7jv88b 2023-12-06T08:09:05.269 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.IEes7jv88b is 317KB 2023-12-06T08:09:05.364 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.IEes7jv88b 2023-12-06T08:09:05.379 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-06T08:09:05.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-06T08:09:05.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-06T08:09:05.766 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T08:09:06.042 INFO:tasks.ceph:Archiving crash dumps... 2023-12-06T08:09:06.044 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi084/crash 2023-12-06T08:09:06.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-06T08:09:06.071 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi107/crash 2023-12-06T08:09:06.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-06T08:09:06.101 INFO:tasks.ceph:Compressing logs... 2023-12-06T08:09:06.101 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T08:09:06.116 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T08:09:06.200 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-06T08:09:06.200 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86567.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86476.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86516.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86567.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86476.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.112201.log 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86476.log.gz 2023-12-06T08:09:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86516.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.109652.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86516.log.gz 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/objectstore_tool.112201.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/objectstore_tool.109652.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.111358.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/objectstore_tool.111358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86396.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86316.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86396.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86607.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86607.log: /var/log/ceph/ceph-client.admin.86316.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86607.log.gz 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86316.log.gz 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-osd.7.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86356.log 2023-12-06T08:09:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86356.log.gz 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mgr.x.log: 94.6% -- replaced with /var/log/ceph/objectstore_tool.112201.log.gz 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr: 94.8% -- replaced with /var/log/ceph/objectstore_tool.109652.log.gzgzip -5 --verbose -- /var/log/ceph/objectstore_tool.110555.log 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr: 94.8% -- replaced with /var/log/ceph/objectstore_tool.111358.log.gz/var/log/ceph/objectstore_tool.110555.log: 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86436.log 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mon.b.log: 88.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.86436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86436.log.gz 2023-12-06T08:09:06.203 INFO:teuthology.orchestra.run.smithi107.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.110555.log.gz 2023-12-06T08:09:06.305 INFO:teuthology.orchestra.run.smithi107.stderr: 90.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-06T08:09:06.325 INFO:teuthology.orchestra.run.smithi107.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-06T08:09:06.331 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124838.log 2023-12-06T08:09:06.331 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128027.log 2023-12-06T08:09:06.331 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124415.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124838.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92601.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128027.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.124415.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.115307.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124415.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130387.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92601.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131379.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115307.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120532.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130387.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92923.log 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131379.log.gz 2023-12-06T08:09:06.332 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92332.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120532.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130203.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92923.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92332.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92332.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130014.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130203.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128308.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91684.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130014.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128308.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.128308.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.110868.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111089.log 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91684.log.gz 2023-12-06T08:09:06.333 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102872.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110868.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115450.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111089.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131253.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102872.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102411.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115450.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97727.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131253.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93231.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102411.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96302.log 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97727.log.gz 2023-12-06T08:09:06.334 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92164.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93231.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95000.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96302.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93554.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92164.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130749.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95000.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114963.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93554.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99027.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130749.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126454.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114963.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90820.log 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99027.log.gz 2023-12-06T08:09:06.335 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129027.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126454.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95597.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90820.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91228.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129027.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128943.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95597.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131001.log 0.0% 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91228.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99068.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128943.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126514.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131001.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90997.log 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99068.log.gz 2023-12-06T08:09:06.336 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111370.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126514.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91429.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90997.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106062.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111370.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106856.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91429.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128733.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106062.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129631.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106856.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120191.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128733.log.gz 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92797.log 2023-12-06T08:09:06.337 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129631.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101970.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120191.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92797.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92797.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101970.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101970.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-06T08:09:06.338 INFO:teuthology.orchestra.run.smithi084.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125205.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/osd.3.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115876.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 64.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120639.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125205.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93641.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115876.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96742.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120639.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99110.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93641.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125605.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96742.log.gz 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92290.log 2023-12-06T08:09:06.339 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99110.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129111.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125605.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110506.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92290.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129330.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129111.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98604.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110506.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116194.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129330.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126874.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98604.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127200.log 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116194.log.gz 2023-12-06T08:09:06.340 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124924.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126874.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127200.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.127200.log.gz --verbose 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.97784.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93109.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124924.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129332.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97784.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96097.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93109.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114450.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129332.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130875.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96097.log.gz 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129575.log 2023-12-06T08:09:06.341 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114450.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129327.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130875.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98169.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129575.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106165.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129327.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128168.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98169.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117621.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106165.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128168.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.128168.log.gz --verbose 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.128607.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94558.log 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117621.log.gz 2023-12-06T08:09:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126535.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128607.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.94558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125062.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94558.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114779.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126535.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103575.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125062.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123057.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114779.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116383.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103575.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122804.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123057.log.gz 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91752.log 2023-12-06T08:09:06.343 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116383.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128481.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122804.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111731.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91752.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117686.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128481.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116321.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111731.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127831.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117686.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114106.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116321.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107491.log 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127831.log.gz 2023-12-06T08:09:06.344 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118164.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114106.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102271.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107491.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90999.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118164.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126032.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102271.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97024.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90999.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101073.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126032.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129069.log 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97024.log.gz 2023-12-06T08:09:06.345 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91274.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101073.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91714.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129069.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130707.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91274.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119224.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91714.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102694.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130707.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93392.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119224.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100690.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102694.log.gz 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.93392.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.126072.log 2023-12-06T08:09:06.346 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93392.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.100690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116197.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100690.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125528.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126072.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124364.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116197.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107533.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125528.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91376.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124364.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123854.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107533.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117865.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91376.log.gz 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114105.log 2023-12-06T08:09:06.347 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123854.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115265.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117865.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114105.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91805.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116157.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115265.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119000.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91805.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93494.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116157.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114408.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119000.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111088.log 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93494.log.gz 2023-12-06T08:09:06.348 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111258.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114408.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106926.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111088.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121877.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111258.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91910.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106926.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110969.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121877.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93110.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91910.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130243.log 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110969.log.gz 2023-12-06T08:09:06.349 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124609.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93110.log.gz 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100591.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130243.log.gz 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131503.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124609.log.gz 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.100591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104677.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100591.log.gz 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107617.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131503.log.gz 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123657.log 2023-12-06T08:09:06.350 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.104677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104677.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121514.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107617.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120104.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123657.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.121514.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91940.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121514.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101580.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120104.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123697.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91940.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.101580.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101580.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108390.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123697.log.gz 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90688.log 2023-12-06T08:09:06.351 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.108390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103313.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108390.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90688.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120043.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107424.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103313.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120043.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112054.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116361.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107424.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112054.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123312.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110828.log 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116361.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123312.log.gz 2023-12-06T08:09:06.352 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110638.log 2023-12-06T08:09:06.353 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130357.log 2023-12-06T08:09:06.353 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110828.log.gz 2023-12-06T08:09:06.353 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110638.log: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-06T08:09:06.354 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110638.log.gz 2023-12-06T08:09:06.354 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116970.log 2023-12-06T08:09:06.354 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129967.log 2023-12-06T08:09:06.354 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130357.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92248.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116970.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99408.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129967.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113719.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92248.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100646.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99408.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97610.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113719.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.100646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117053.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100646.log.gz 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123008.log 2023-12-06T08:09:06.355 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97610.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101136.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117053.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91654.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123008.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101136.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97923.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118984.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91654.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124445.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97923.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106749.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118984.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92416.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124445.log.gz 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112771.log 2023-12-06T08:09:06.356 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106749.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101704.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92416.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128392.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112771.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101704.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101704.log.gz -5 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92598.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129331.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128392.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108570.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92598.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120642.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129331.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.108570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131211.log 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108570.log.gz 2023-12-06T08:09:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107648.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120642.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118250.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131211.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121554.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107648.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125743.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118250.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130088.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121554.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120328.log 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125743.log.gz 2023-12-06T08:09:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112434.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130088.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103658.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120328.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130581.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112434.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113555.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103658.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121196.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130581.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115663.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113555.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102653.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121196.log.gz 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118872.log 2023-12-06T08:09:06.359 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115663.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89033.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102653.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127475.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118872.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.89033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98112.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89033.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124301.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127475.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106466.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98112.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124301.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124301.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90778.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106466.log.gz 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126333.log 2023-12-06T08:09:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94873.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90778.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114617.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126333.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.94873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94873.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115816.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95043.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114617.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115816.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111153.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95452.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95043.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111153.log.gz 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128565.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124527.log 2023-12-06T08:09:06.361 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95452.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128565.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91870.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92206.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124527.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91870.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.91870.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.111556.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120450.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92206.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90994.log 0.0% 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111556.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131421.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120450.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90994.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90994.log.gz 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90588.log 2023-12-06T08:09:06.362 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118048.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131421.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94156.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90588.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95402.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118048.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.94156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102150.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94156.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91603.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95402.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102150.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.113899.log -- replaced with /var/log/ceph/ceph-client.admin.102150.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86697.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91603.log.gz 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113899.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.113899.log.gz -- 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.122543.log 2023-12-06T08:09:06.363 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.86697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128691.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86697.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122543.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128350.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118528.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128691.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90318.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128350.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.118528.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.98634.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118528.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130263.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90318.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87935.log 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98634.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130263.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.130263.log.gz 2023-12-06T08:09:06.364 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.114365.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.87935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120955.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87935.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131295.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114365.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107311.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120955.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131295.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.131295.log.gz /var/log/ceph/ceph-client.admin.107162.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114491.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107311.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123997.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107162.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.114491.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.131043.log 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114491.log.gz 2023-12-06T08:09:06.365 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129626.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123997.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131043.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127335.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129626.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118757.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118394.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127335.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111875.log 0.0% 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118757.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122651.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118394.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97548.log 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111875.log.gz 2023-12-06T08:09:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87026.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122651.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97548.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117013.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.87026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114705.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87026.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117013.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93048.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90644.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114705.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114020.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93048.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111603.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90644.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127786.log 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114020.log.gz 2023-12-06T08:09:06.367 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111603.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118209.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125991.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127786.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118209.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.118209.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.124221.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102152.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125991.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107575.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124221.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102152.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102152.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130497.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107575.log.gz 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126945.log 2023-12-06T08:09:06.368 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131085.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130497.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113818.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126945.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118633.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131085.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98728.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113818.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118633.log.gzgzip 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.128859.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100895.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98728.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129153.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128859.log.gz 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.100895.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100895.log.gz -5 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.125487.log 2023-12-06T08:09:06.369 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90533.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129153.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125487.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131127.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124142.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90533.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131127.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127746.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101811.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124142.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127746.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123503.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129907.log 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101811.log.gz 2023-12-06T08:09:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123503.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130959.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115732.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129907.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130959.log: gzip 26.6% -- replaced with /var/log/ceph/ceph-client.admin.130959.log.gz -5 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.99465.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115531.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115732.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99465.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99465.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.118717.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115531.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112196.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92009.log 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118717.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112196.log.gz 2023-12-06T08:09:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80846.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115989.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92009.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.80846.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80846.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119765.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114533.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115989.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119765.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91534.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98966.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114533.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91534.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113656.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131169.log 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98966.log.gz 2023-12-06T08:09:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113656.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105998.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131463.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131169.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.105998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105998.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91189.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119274.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.131463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131463.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91189.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130791.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121176.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119274.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117519.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130791.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128901.log 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121176.log.gz 2023-12-06T08:09:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117519.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122458.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94679.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128901.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117135.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122458.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.94679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111975.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94679.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117135.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125364.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125574.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.111975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111975.log.gz 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125364.log.gzgzip 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.119363.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114648.log 2023-12-06T08:09:06.374 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125574.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130539.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119363.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96138.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114648.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114575.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130539.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107267.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96138.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114575.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117416.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124692.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107267.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117416.log.gz 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122587.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103428.log 2023-12-06T08:09:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124692.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122587.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93174.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103617.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103428.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93174.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.93174.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.123571.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100979.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103617.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117336.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123571.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.100979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103449.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100979.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116498.log 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117336.log.gz 2023-12-06T08:09:06.376 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103449.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103449.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103094.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116498.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96404.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121016.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103094.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128817.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96404.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91267.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121016.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121614.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128817.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90995.log 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91267.log.gz 2023-12-06T08:09:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.121614.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121614.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.114758.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90995.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92478.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114758.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.107711.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114758.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122502.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92478.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129701.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107711.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125890.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122502.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110589.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129701.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103193.log 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125890.log.gz 2023-12-06T08:09:06.378 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.110589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96662.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110589.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129570.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103193.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86999.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96662.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118334.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129570.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.86999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96364.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86999.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120875.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118334.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.96364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98268.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96364.log.gz 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108093.log 2023-12-06T08:09:06.379 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120875.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129325.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.98268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98268.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.108093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124028.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108093.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107194.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129325.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112731.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124028.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117763.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107194.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95920.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112731.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91980.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117763.log.gz 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125022.log 2023-12-06T08:09:06.380 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95920.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130833.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91980.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.125022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91316.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125022.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90992.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130833.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130665.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 0.0% 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91316.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127113.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90992.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130917.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130665.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127113.log.gz 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129966.log 2023-12-06T08:09:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129867.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130917.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101791.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129966.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129867.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114323.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130108.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101791.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.114323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120246.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114323.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130108.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91001.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120246.log.gz 2023-12-06T08:09:06.382 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123614.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91001.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108052.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120552.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123614.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.108052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126168.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108052.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122947.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120552.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99655.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126168.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116453.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122947.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128439.log 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99655.log.gz 2023-12-06T08:09:06.383 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93006.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116453.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130010.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128439.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127634.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93006.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130010.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130010.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.93774.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.127634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127634.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92374.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93774.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.93774.log.gz 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.115166.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92374.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.92374.log.gz -- 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.101435.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115166.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.115166.log.gz -- 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr: /var/log/ceph/ceph-client.admin.95225.log 2023-12-06T08:09:06.384 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92599.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101435.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87141.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95225.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91002.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92599.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.87141.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87141.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117601.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91002.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101311.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107861.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117601.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129334.log 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101311.log.gz 2023-12-06T08:09:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91258.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107861.log: /var/log/ceph/ceph-client.admin.129334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123392.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129334.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107861.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.91258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91258.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124732.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123392.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124141.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.123392.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124732.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124732.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115772.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124141.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: 93.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101662.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95834.log 2023-12-06T08:09:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115772.log: /var/log/ceph/ceph-client.admin.101662.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129326.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.115772.log.gz 0.0% 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101662.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116811.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95834.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102528.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129326.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116811.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116811.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.129700.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.102528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102528.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116667.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129700.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129335.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118116.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116667.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.129335.log: -- /var/log/ceph/ceph-client.admin.123078.log 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116667.log.gz 2023-12-06T08:09:06.387 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129335.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.118116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124980.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118116.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130623.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123078.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124980.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.103054.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.124980.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130623.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119682.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128649.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.103054.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.119682.log: /var/log/ceph/ceph-client.admin.106380.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103054.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119682.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128649.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128649.log.gz 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.129580.log 2023-12-06T08:09:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106380.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94759.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.129580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129580.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113758.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.94759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94759.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117173.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.113758.log: -- /var/log/ceph/ceph-client.admin.128985.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113758.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117173.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99868.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128985.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113013.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99868.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99868.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.112216.log 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113013.log.gz 2023-12-06T08:09:06.389 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92048.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.112216.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112216.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.130455.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92048.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128210.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.130455.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.125142.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.130455.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126814.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128210.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5/var/log/ceph/ceph-client.admin.125142.log: --verbose -- /var/log/ceph/ceph-client.admin.90736.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125142.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.126814.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101353.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126814.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5/var/log/ceph/ceph-client.admin.90736.log: --verbose -- /var/log/ceph/ceph-client.admin.128775.log 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90736.log.gz 2023-12-06T08:09:06.390 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101353.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93894.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128775.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101600.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.93894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93894.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98476.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101600.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128266.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.98476.log: -- /var/log/ceph/ceph-client.admin.131337.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98476.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128266.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106423.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.131337.log: -- /var/log/ceph/ceph-client.admin.101749.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131337.log.gz 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106423.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129627.log 2023-12-06T08:09:06.391 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106423.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.101749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113335.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101749.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5/var/log/ceph/ceph-client.admin.129627.log: --verbose -- /var/log/ceph/ceph-client.admin.128523.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129627.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.113335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113335.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119724.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.128523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128523.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123169.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119724.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91555.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118798.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.123169.log: gzip/var/log/ceph/ceph-client.admin.91555.log: -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.116607.log 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91555.log.gz 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123169.log.gz/var/log/ceph/ceph-client.admin.118798.log: 2023-12-06T08:09:06.392 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118798.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90998.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.116607.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.95900.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116607.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.90998.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90998.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.115219.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95900.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97385.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115219.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124773.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.97385.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97385.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.99615.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.124773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124773.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117561.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.99615.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99615.log.gz 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.107100.log 2023-12-06T08:09:06.393 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.117561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117561.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109212.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.107100.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.130323.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107100.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.109212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120145.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109212.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.130323.log: -- /var/log/ceph/ceph-client.admin.92080.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130323.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.120145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120145.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101498.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122299.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.92080.log: /var/log/ceph/ceph-client.admin.101498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95700.log 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92080.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101498.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.122299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122299.log.gz 2023-12-06T08:09:06.394 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116870.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115106.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.95700.log: /var/log/ceph/ceph-client.admin.116870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119485.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95700.log.gz 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116870.log.gz 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.115106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115106.log.gz 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114283.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106539.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.119485.log: /var/log/ceph/ceph-client.admin.114283.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119485.log.gz 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114283.log.gz 2023-12-06T08:09:06.395 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.106539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106539.log.gz 2023-12-06T08:09:06.396 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-06T08:09:06.396 INFO:teuthology.orchestra.run.smithi084.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-06T08:09:06.396 INFO:teuthology.orchestra.run.smithi084.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-06T08:09:06.397 INFO:teuthology.orchestra.run.smithi107.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-06T08:09:06.397 INFO:teuthology.orchestra.run.smithi107.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-06T08:09:06.397 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-06T08:09:06.405 INFO:teuthology.orchestra.run.smithi084.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-06T08:09:06.486 INFO:teuthology.orchestra.run.smithi107.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-06T08:09:06.490 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-12-06T08:09:06.490 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.328s 2023-12-06T08:09:06.490 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m1.108s 2023-12-06T08:09:06.490 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.068s 2023-12-06T08:09:06.504 INFO:teuthology.orchestra.run.smithi084.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-06T08:09:06.574 INFO:teuthology.orchestra.run.smithi084.stderr: 89.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-06T08:09:06.576 INFO:teuthology.orchestra.run.smithi084.stderr: 2023-12-06T08:09:06.576 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m0.449s 2023-12-06T08:09:06.576 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m1.416s 2023-12-06T08:09:06.576 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.237s 2023-12-06T08:09:06.576 INFO:tasks.ceph:Archiving logs... 2023-12-06T08:09:06.576 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi084/log 2023-12-06T08:09:06.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T08:09:07.243 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi107/log 2023-12-06T08:09:07.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T08:09:07.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-06T08:09:07.482 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T08:09:07.483 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-06T08:09:07.498 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_96fa2578c622ca9c169af6f426ca8b1616130e58/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T08:09:07.499 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-06T08:09:07.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T08:09:07.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T08:09:07.583 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-06T08:09:07.592 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-06T08:09:07.592 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T08:09:07.595 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T08:09:07.607 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2023-12-06T08:09:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 +1205us[+1213us] +/- 90ms 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -922us[ -915us] +/- 61ms 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +257us[ +264us] +/- 103ms 2023-12-06T08:09:07.611 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 38 +1216us[+1223us] +/- 90ms 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -944us[ -937us] +/- 61ms 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +246us[ +246us] +/- 103ms 2023-12-06T08:09:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T08:09:07.615 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-06T08:09:07.624 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-06T08:09:07.625 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-06T08:09:07.651 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T08:09:07.704 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T08:09:07.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T08:09:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T08:09:07.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T08:09:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T08:09:07.874 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-06T08:09:07.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-06T08:09:07.942 INFO:teuthology.task.internal:Duration was 1743.942237 seconds 2023-12-06T08:09:07.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-06T08:09:07.978 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-06T08:09:07.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T08:09:07.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T08:09:08.021 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T08:09:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T08:09:08.276 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-06T08:09:08.276 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2023-12-06T08:09:08.277 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T08:09:08.303 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-12-06T08:09:08.304 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T08:09:08.337 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-06T08:09:08.337 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T08:09:08.346 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T08:09:08.406 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-06T08:09:08.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T08:09:08.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T08:09:08.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-06T08:09:08.586 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-06T08:09:08.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T08:09:08.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T08:09:08.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-06T08:09:08.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T08:09:08.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T08:09:08.679 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2023-12-06T08:09:08.706 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-12-06T08:09:08.726 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T08:09:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T08:09:08.763 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T08:09:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T08:09:08.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-06T08:09:08.793 INFO:teuthology.task.internal:Transferring archived files... 2023-12-06T08:09:08.794 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi084 2023-12-06T08:09:08.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T08:09:08.836 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417/remote/smithi107 2023-12-06T08:09:08.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T08:09:08.874 INFO:teuthology.task.internal:Removing archive directory... 2023-12-06T08:09:08.874 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T08:09:08.877 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T08:09:08.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-06T08:09:08.935 INFO:teuthology.task.internal:Not uploading archives. 2023-12-06T08:09:08.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-06T08:09:08.974 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-06T08:09:08.974 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T08:09:08.977 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T08:09:08.989 INFO:teuthology.orchestra.run.smithi084.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 08:09 /home/ubuntu/cephtest 2023-12-06T08:09:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: 265818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 08:09 /home/ubuntu/cephtest 2023-12-06T08:09:08.992 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-06T08:09:09.003 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-06T08:09:09.039 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-06T08:09:09.039 INFO:teuthology.nuke:Checking targets against current locks 2023-12-06T08:09:09.060 DEBUG:teuthology.nuke:shortname: smithi084 2023-12-06T08:09:09.060 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T08:09:09.085 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 07:31:26.823882', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQVssVISaQqOamGjBZt/mY8Q2tIklX0RUy5g1WQZzGogN+AXO2/DfEhuPAJ7R8ULDgF4tL4li1qOe2kmKwf8BQ='} 2023-12-06T08:09:09.090 DEBUG:teuthology.nuke:shortname: smithi107 2023-12-06T08:09:09.090 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T08:09:09.113 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-12-06_06:16:39-rados-wip-sseshasa-testing-2023-12-04-1848-distro-default-smithi/7479417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 07:31:26.825339', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvIqCUOjcCj8G+G4cMbYPHfTHYBX87fiDKJMbdetC3yteJVYM9dkUHa/psN7FRl95bRJ+DNpXqfLM5Zv/zGt+M='} 2023-12-06T08:09:09.137 INFO:teuthology.orchestra.console.smithi084:Power off 2023-12-06T08:09:09.138 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T08:09:09.166 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2023-12-06T08:09:09.166 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:09.185 INFO:teuthology.orchestra.console.smithi107:Power off 2023-12-06T08:09:09.185 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T08:09:09.195 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2023-12-06T08:09:09.211 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-12-06T08:09:09.211 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:09.235 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-06T08:09:13.197 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:13.236 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:13.311 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2023-12-06T08:09:13.346 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-06T08:09:17.316 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:17.347 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T08:09:17.428 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2023-12-06T08:09:17.458 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-12-06T08:09:17.529 INFO:teuthology.orchestra.console.smithi084:Power off completed 2023-12-06T08:09:17.560 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-12-06T08:09:17.664 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} duration: 1743.942236661911 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_sseshasa@teuthology status: fail success: false 2023-12-06T08:09:17.665 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T08:09:17.751 INFO:teuthology.run:FAIL